python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3490622 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 8562>200 please execute split_portfolio.py -i 23145208 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2900254'] with mtr_portfolio_ids : ['23145208'] and first list_photo_ids : [] new path : /proc/3490622/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 34.6796875 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon May 26 12:11:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-05-26 12:11:13.860114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-26 12:11:13.891465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 12:11:13.891744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 12:11:13.897401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 12:11:13.899515: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 12:11:13.899877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 12:11:13.902601: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 12:11:13.903929: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 12:11:13.909033: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 12:11:13.910273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 12:11:13.910608: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-26 12:11:13.935256: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-26 12:11:13.937005: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f72dc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-26 12:11:13.937058: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-26 12:11:14.076310: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x393b3bb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-26 12:11:14.076374: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-26 12:11:14.077252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 12:11:14.077338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 12:11:14.077361: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 12:11:14.077382: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 12:11:14.077403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 12:11:14.077423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 12:11:14.077443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 12:11:14.077464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 12:11:14.078488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 12:11:14.078536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 12:11:14.079189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 12:11:14.079208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 12:11:14.079218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 12:11:14.080301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2855 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-26 12:11:21.221041: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-26 12:11:21.221706: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.484474420547485 time used to load_weights : 0.2895216941833496 0it [00:00, ?it/s] 58it [00:00, 577.33it/s]temp/1748254231_3490622_1361016673_638c18e48aa097bc71c75486efdc3aad.jpg temp/1748254231_3490622_1361016671_97d8954c90f422b0260a5dacfe921f14.jpg temp/1748254231_3490622_1361016640_ea089640c2517d0200957e8cb7af95c7.jpg temp/1748254231_3490622_1361016639_25973dbc38a5b2e2da3bf27f2c8642b1.jpg temp/1748254231_3490622_1361016638_583446ec8e632081ddeb41e99002eadf.jpg temp/1748254231_3490622_1361016637_79c66c92a8e38fe957a1dae713595e31.jpg temp/1748254231_3490622_1361016636_b73a492d7c42bf71e1a6667cf63360bb.jpg temp/1748254231_3490622_1361016634_40ecd734a2e424fa405930617d364709.jpg temp/1748254231_3490622_1361016603_44e215bf20470bd147387a2fe00ee863.jpg temp/1748254231_3490622_1361016602_5e065b0a945ecab7aafeb89a2655c622.jpg temp/1748254231_3490622_1361016600_c7f218137c38c96fa98b0ef00b728ca3.jpg temp/1748254231_3490622_1361016598_91e1f1a33c26b218a3adcee49ad3de2d.jpg temp/1748254231_3490622_1361016596_285e728160427ecc7ae48c020c24aec5.jpg temp/1748254231_3490622_1361016594_6c31336c6d3e64846cd69516e31eb5c0.jpg temp/1748254231_3490622_1361016220_83bbc7bb7532dd5934e76c3b28a45653.jpg temp/1748254231_3490622_1361016219_435dc41725c74035406438faf1f2512f.jpg temp/1748254231_3490622_1361016218_20bf3b95fca456933d07111377dbd63b.jpg temp/1748254231_3490622_1361016216_21ccf56393cf4c97f94396a1045740cf.jpg temp/1748254231_3490622_1361016213_02eefa0c460edddfde2baf366475f527.jpg temp/1748254231_3490622_1361016210_970ced95dfdd7d7612d813a36defc0b2.jpg temp/1748254231_3490622_1361016167_fd3a9af6a38a8a42a50836f1ddd341e3.jpg temp/1748254231_3490622_1361016166_affe10ec802d1eed8063579d6afbd3f4.jpg temp/1748254231_3490622_1361016164_113050b6109f64c0c7a9ecad946eb1de.jpg temp/1748254231_3490622_1361016163_1273b831180b3320131bad72384600a1.jpg temp/1748254231_3490622_1361016162_929838cd7e0924d1a7eec65bc40fdd5a.jpg temp/1748254231_3490622_1361016161_d93b2d0413c7f0f2571e003a31146a51.jpg temp/1748254231_3490622_1361016072_a8899a4d3032e2cbb83a3453e488a095.jpg temp/1748254231_3490622_1361016071_8c19477cbd28247b0e491b4703ee8047.jpg temp/1748254231_3490622_1361016070_b765eb88132ba4c39c8278da65dfec00.jpg temp/1748254231_3490622_1361016069_63f0b5bdcfdad569b67b3a480aa81b85.jpg temp/1748254231_3490622_1361016068_58a5a87db69a0b8163bd0c25e3c12675.jpg temp/1748254231_3490622_1361016065_a6e7fd58270e73741d1b175c72830082.jpg temp/1748254231_3490622_1361016007_15075c51c141d883fc43b8fdc0c14ea9.jpg temp/1748254231_3490622_1361016006_569c968571e05a8990be7bef9e5b1d1d.jpg temp/1748254231_3490622_1361016005_a33560e1cac2c7c7ad9d0f347b7de451.jpg temp/1748254231_3490622_1361016004_3e28b4380541f77afb18d9fd4542256d.jpg temp/1748254231_3490622_1361016003_d79c20d44b7762afeffb020386dd5b3d.jpg temp/1748254231_3490622_1361016001_d724b929372bdac318a17772132422b0.jpg temp/1748254231_3490622_1361015847_b3681bdb10e77252e8add190e223ba43.jpg temp/1748254231_3490622_1361015846_d4630c50bf47a6a274132d36fdf45ee0.jpg temp/1748254231_3490622_1361015845_a722c9b2b0f58b272adff17f6731c814.jpg temp/1748254231_3490622_1361015844_b8f3d69edc7bafac2805d3fbb4211c57.jpg temp/1748254231_3490622_1361015843_65231f5df740a681258b4442d5e63866.jpg temp/1748254231_3490622_1361015841_45a44732226607a239ef12a7e48a6338.jpg temp/1748254231_3490622_1361015699_d810d6c6f2ae31a4b0596f62e0193c2b.jpg temp/1748254231_3490622_1361015698_dc9f0884c17270444cefdcb1bc3e8045.jpg temp/1748254231_3490622_1361015697_8e1316bd574bc5587558f5112aa3a124.jpg temp/1748254231_3490622_1361015696_e1a7ff4df12a8102bd168674ae2d4e2d.jpg temp/1748254231_3490622_1361015694_f50e7c2f1aea447cf83e05657bf06ce3.jpg temp/1748254231_3490622_1361015691_faa04d6185d640ca463c5bfcda5338cc.jpg temp/1748254231_3490622_1361015619_5980f5f0dd6d2738448758fea8c4a451.jpg temp/1748254231_3490622_1361015618_1f3d251d94f368a5580109b1a63fc075.jpg temp/1748254231_3490622_1361015616_ec96701f32025880886e7c7defa2225a.jpg temp/1748254231_3490622_1361015614_a9e0bf49fa4a8d866dfcb1e12d108155.jpg temp/1748254231_3490622_1361015612_cd3668cf030f0e16a392d38373fe5960.jpg temp/1748254231_3490622_1361015609_9bf4179e45499383039d89233f41de51.jpg temp/1748254231_3490622_1361015554_76f5f0f823b307708b80059fea01de0f.jpg temp/1748254231_3490622_1361015553_d8d8bd6a2ba34c58969f155fd8179b15.jpg temp/1748254231_3490622_1361015552_6f5f0502f41e70f065d6aba895e3297c.jpg temp/1748254231_3490622_1361015551_fc05861ef27ee3c90d230f9e739b103c.jpg temp/1748254231_3490622_1361015550_36d7045de0f01272b01cd04b6b40f686.jpg temp/1748254231_3490622_1361015548_e3a302c06e4c373e18a555e2c78cbf1a.jpg temp/1748254231_3490622_1361015516_8996a14d333bd1816f515b3a4d63fb52.jpg temp/1748254231_3490622_1361015515_3304aedb4dbd1ceb8218272ff93a7680.jpg temp/1748254231_3490622_1361015514_7be1fd98183fce790e83fa3cb00540f7.jpg temp/1748254231_3490622_1361015512_529298b4bef08882eaeddf4fb14c4494.jpg temp/1748254231_3490622_1361015510_1534eeaa95e91fd9324a09406d8605fc.jpg temp/1748254231_3490622_1361015508_c88925c98713614d716f287e2baf5d1d.jpg temp/1748254231_3490622_1361015402_a2a928057ed792e524b66316505366f9.jpg temp/1748254231_3490622_1361015401_aa96fc906125a52137fe1f18af2795f9.jpg temp/1748254231_3490622_1361015400_40dd544fb509183d5d2aeb20536543d2.jpg temp/1748254231_3490622_1361015399_b150f3a61c72571ac844b931f02685d9.jpg temp/1748254231_3490622_1361015397_a1307b5f2126569dd765b4b581256d24.jpg temp/1748254231_3490622_1361015395_1356254cdbab218e3fb0d01d2e8bcf2b.jpg temp/1748254231_3490622_1361015042_d08aabcd3aeaf406668fbb60892124d2.jpg temp/1748254231_3490622_1361015041_14d762071cfdcc31577faa039badb9cc.jpg temp/1748254231_3490622_1361015039_acdeb390fedcc8f8839dfad84c90641f.jpg temp/1748254231_3490622_1361015037_49d9a7067440dbfcfa03214a217e3ffd.jpg temp/1748254231_3490622_1361015035_c7f3129f2afbf59f0c1a9be2e49aaed0.jpg temp/1748254231_3490622_1361015033_e4ce07e75098caa189c722e31c346e17.jpg temp/1748254231_3490622_1361014912_b287f6810170d43c53146166f79c8c9a.jpg temp/1748254231_3490622_1361014911_4596a416068be2f614a8fcd071268649.jpg temp/1748254231_3490622_1361014909_62b9fbf1329601c92fa19b462d2968a8.jpg temp/1748254231_3490622_1361014907_cb5bd25e65551185acabfb9bd4817744.jpg temp/1748254231_3490622_1361014905_c9b26f0e8279b02bb70297e643d34d74.jpg temp/1748254231_3490622_1361014904_6909a4cbaa12a6abae6f06ace9d0eaa2.jpg temp/1748254231_3490622_1361014858_88a5423aeaf65f5e352cbe07f47b16b6.jpg temp/1748254231_3490622_1361014857_0e866c5a05561bb329dccdab1ae3880d.jpg temp/1748254231_3490622_1361014856_2d9910d5fe7630daf40ff7b0079d4167.jpg temp/1748254231_3490622_1361014855_4212544fad6be5fc2ab727c7cf5d6f25.jpg temp/1748254231_3490622_1361014854_8deab90d0a1b6915f9d95e09513c394b.jpg temp/1748254231_3490622_1361014852_2ac1b1d3cf345d75c6899ebdaea6c2bc.jpg temp/1748254231_3490622_1361014789_f926556df2f079254ee63e1c185add3b.jpg temp/1748254231_3490622_1361014788_36b4b4696446c7efdd904a1c98d26c3d.jpg temp/1748254231_3490622_1361014786_ac3a9bdc7897f2d28e2bcd5da4ee855d.jpg temp/1748254231_3490622_1361014784_37569fa4a8e183089339ec4cf63d41aa.jpg temp/1748254231_3490622_1361014782_8fbbb5e6300be2cfb103242e74dbf64b.jpg temp/1748254231_3490622_1361014780_322d025ecd1ff2ccf17bdf1c9dc02835.jpg temp/1748254231_3490622_1361014694_ecd03e64d9ba617c9e80e9c2fdd81c4e.jpg temp/1748254231_3490622_1361014693_d47f60e7e4dab42b02e7f304629aa740.jpg temp/1748254231_3490622_1361014691_1d61cda714b84169b8cba63e213ddaf0.jpg temp/1748254231_3490622_1361014689_ca8a1fc9a664fa5012ee25eb2194df55.jpg temp/1748254231_3490622_1361014687_44651a46a2a8249c735ba39c1e95e507.jpg temp/1748254231_3490622_1361014685_a983d14d573213ef294df6c17fe3df6c.jpg temp/1748254231_3490622_1361014437_8b671dee37d2b91f63cd8c516efdc5d0.jpg temp/1748254231_3490622_1361014436_9d4441c41baf0b2e3ee85bb3db4478b3.jpg temp/1748254231_3490622_1361014435_35bf6fa7dd4524c11d542be6db143c17.jpg temp/1748254231_3490622_1361014433_1ce0e0754814ad24e92514a01fd0db7e.jpg temp/1748254231_3490622_1361014432_21a2bad7366bd7be6bce0f6b2ab26650.jpg temp/1748254231_3490622_1361014430_9896df15e27c98709914864df119f783.jpg temp/1748254231_3490622_1361014325_f68618f2eef6f893ccf1a5959c7569cb.jpg temp/1748254231_3490622_1361014323_d3df82af699c7e23ff084f5ece5697c1.jpg temp/1748254231_3490622_1361014321_508500284beb0609aaab3e51426822b7.jpg 122it [00:00, 613.69it/s] 200it [00:00, 687.50it/s] 200it [00:00, 663.66it/s]2025-05-26 12:11:25.477707: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 12:11:25.597044: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-26 12:11:25.622502: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 12:11:25.635434: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-26 12:11:25.640537: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748254231_3490622_1361014319_41e2c334b966043acab507d46a994461.jpg temp/1748254231_3490622_1361014317_7d28a3ee800d6a9fb430dfe2adce4b06.jpg temp/1748254231_3490622_1361014315_f2bff0c556a71f4030610ee5110f6a30.jpg temp/1748254231_3490622_1361014214_5f29c348f7bbdb17301e59a5fc6e4352.jpg temp/1748254231_3490622_1361014212_156b7234543dd79749ed485197fbad8d.jpg temp/1748254231_3490622_1361014211_bff780a45bbddbd009e637cb2f64856b.jpg temp/1748254231_3490622_1361014210_89b9129ab89b7c25350c5551513803cc.jpg temp/1748254231_3490622_1361014209_16a39fcdeed61eed94cf8597136e6457.jpg temp/1748254231_3490622_1361014206_546e48552ddd61049f21fbd95afb297d.jpg temp/1748254231_3490622_1361014013_6a1151f49a4b826fe56041d05f7d4437.jpg temp/1748254231_3490622_1361014012_579c59b96a44a3f01f57300d30262e59.jpg temp/1748254231_3490622_1361014011_94fae70fbb7773b2e8322b4c26216a67.jpg temp/1748254231_3490622_1361014010_88fa143076023d6f25987385d24a2e1c.jpg temp/1748254231_3490622_1361014007_4ecf10435ed0fe694b45b0efb179cc64.jpg temp/1748254231_3490622_1361014004_0b24daca4bbe37c30330d64ce56f6803.jpg temp/1748254231_3490622_1361013851_47be8f3690044a795ff45d83628d869e.jpg temp/1748254231_3490622_1361013850_b58f07765794e27332605126f6574275.jpg temp/1748254231_3490622_1361013849_91fb948f4683d34daca61bb9ca028b8b.jpg temp/1748254231_3490622_1361013848_686df0acecbd4779768836c1e16d546f.jpg temp/1748254231_3490622_1361013846_74f7c7b642fbca4dd745fb2ada3e306f.jpg temp/1748254231_3490622_1361013843_81f0e62f91e7905fe36d708fd266a2f6.jpg temp/1748254231_3490622_1361013738_82ea394d73559ff3c4f85be8513be0fb.jpg temp/1748254231_3490622_1361013737_73010eedb20d6223d169f10cc9eef006.jpg temp/1748254231_3490622_1361013736_64e0f0971fb29db8d534eb103ced8c24.jpg temp/1748254231_3490622_1361013735_f6cff348c5a6b9ffc374783d375f8091.jpg temp/1748254231_3490622_1361013733_4f05dd1ebdb9a35db03ae2459e5cf1f5.jpg temp/1748254231_3490622_1361013732_9d27a5b3a6eb40ea9992db4712ce8c37.jpg temp/1748254231_3490622_1361012396_56fbee10c636df8443d089e4edbfce10.jpg temp/1748254231_3490622_1361012392_144359667e8c2b5d1b711d3e647c1aed.jpg temp/1748254231_3490622_1361012391_15541684dfa6859240bce18c0af783cb.jpg temp/1748254231_3490622_1361012389_ddd20187b1da6a36a7d0dd569969f6c9.jpg temp/1748254231_3490622_1361012387_39aa523218d4e91b75855e2271b7443d.jpg temp/1748254231_3490622_1361012385_aaa873a6b1160f4fc22308f55315bd97.jpg temp/1748254231_3490622_1361011805_96a0f0813f8da4d60d5bbecb6b5f4654.jpg temp/1748254231_3490622_1361011779_2d97344b609696296ea2cfa35cee7005.jpg temp/1748254231_3490622_1361011753_4289695f3bd9340a22a66be345722a39.jpg temp/1748254231_3490622_1361011725_91fa520998f7a0e5e0dac8f32440e77b.jpg temp/1748254231_3490622_1361011699_a0238035ea46b258d0563d980394342d.jpg temp/1748254231_3490622_1361011669_891689c4ebfa5172f451ed5697e76585.jpg temp/1748254231_3490622_1361011121_f2cdeafcd02abbfbbdfc9af78354af71.jpg temp/1748254231_3490622_1361011120_eecd4cfc8e9cd0d831fd283b9c24204f.jpg temp/1748254231_3490622_1361011119_883902bd892c7256f37f514a601badf4.jpg temp/1748254231_3490622_1361011118_1340065ec5f83a35292c1945364419af.jpg temp/1748254231_3490622_1361011116_8bc9074706a4581ab4a4ddded5577cca.jpg temp/1748254231_3490622_1361011114_2a1fc1e009fba0b82836084b096d78da.jpg temp/1748254231_3490622_1361011103_c3beb2c65b68830e47bf26ebd6f3146e.jpg temp/1748254231_3490622_1361011102_8995b45117cc09897a55681cef637d8b.jpg temp/1748254231_3490622_1361011101_f7b3262723a6303f709456393524dd83.jpg temp/1748254231_3490622_1361011100_a32759f46c0d2fffc8241501175a53c9.jpg temp/1748254231_3490622_1361011098_95ece1ca0bae9489892daa52e8375eb9.jpg temp/1748254231_3490622_1361011095_a41b1e05e5da44a6bcde6e9692aebe3b.jpg temp/1748254231_3490622_1361011072_3c76d22845131504c178dbd0deb2480d.jpg temp/1748254231_3490622_1361011045_53a4f799b1f9aa44210f6b6f02811d6e.jpg temp/1748254231_3490622_1361011032_7a54d237d0f7bd5512c405391a5b470e.jpg temp/1748254231_3490622_1361011031_b536891bc378e68f47810950eaafce0d.jpg temp/1748254231_3490622_1361011018_5c611800f08f41def2e3cf099ba19b77.jpg temp/1748254231_3490622_1361010988_a815435c1243fe16f2769b42a8d547e7.jpg temp/1748254231_3490622_1361006409_4b6e8b7d84d94194b002310b9f2806e8.jpg temp/1748254231_3490622_1361006383_310c7faaeacea9218f02ae3d9cdfdfc1.jpg temp/1748254231_3490622_1361006357_fbf680942dc6bfa64d26fba1a277a4b3.jpg temp/1748254231_3490622_1361006354_585356c72aba191e18d5abaa203cd98a.jpg temp/1748254231_3490622_1361006351_4c39dee7038b0a518b5c9d1773b511c0.jpg temp/1748254231_3490622_1361006349_a9575c542b9d2dba1f7cf7c07abbd33c.jpg temp/1748254231_3490622_1361003459_672dd61d6e8365f0c8ab50814245f6c3.jpg temp/1748254231_3490622_1361003458_3eaf3ae8fea3fba9820dfd8aa3fd496b.jpg temp/1748254231_3490622_1361003457_55987e4ab774322a24aa31a2e0293859.jpg temp/1748254231_3490622_1361003456_f6c784f1e73059594153df45bb4e14b6.jpg temp/1748254231_3490622_1361003454_3419ad1536a072de2d23a5d3e973bb57.jpg temp/1748254231_3490622_1361003452_d0a48bfaaf56fa23eb44cae1de08373d.jpg temp/1748254231_3490622_1361003110_43c49d4d595da3c7d68dc798c3c1dd52.jpg temp/1748254231_3490622_1361003084_028046b2dec090c8546b2fc2ca5b49a7.jpg temp/1748254231_3490622_1361003057_8e9a3af9f34586e90bf3bd5485825e41.jpg temp/1748254231_3490622_1361003031_95b79430310dfa3b182167d5c4346e3d.jpg temp/1748254231_3490622_1361003004_13149f6ef8e41abd95fa1bd0cf2c0c0a.jpg temp/1748254231_3490622_1361002977_02c94b52a24b5971ab2ea6b197becbbf.jpg temp/1748254231_3490622_1361002622_9d7838efd0efc918f2a8336e42530fad.jpg temp/1748254231_3490622_1361002591_7c7267c21ef7064854622bac68421f5c.jpg temp/1748254231_3490622_1361002563_9340c322d10fcc3bc4a1021b9fbcab7d.jpg temp/1748254231_3490622_1361002534_21ad49cd9e98a67064fd8527bdbd1bb3.jpg temp/1748254231_3490622_1361002507_3248042f8ad47ab0cdab29a0fcca91db.jpg temp/1748254231_3490622_1361002480_7379b3daf3ef06d4cabc5bf6779b87e7.jpg temp/1748254231_3490622_1361002129_da0de20dfb223dae4daa6c78e9156c54.jpg temp/1748254231_3490622_1361002128_a3f540ab66d693ab28e5a0749e9f4e50.jpg temp/1748254231_3490622_1361002127_489a3556589596d8c7f1310f85c9c641.jpg temp/1748254231_3490622_1361002126_b206c5e454a401f73d2c6f6abf48ca99.jpg temp/1748254231_3490622_1361002125_f1253c48cbcfbdaedcde48465667f023.jpg temp/1748254231_3490622_1361002121_6bd08f2f953a358ffec677e28a43a3d2.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361016673, 1361016671, 1361016640, 1361016639, 1361016638, 1361016637, 1361016636, 1361016634, 1361016603, 1361016602, 1361016600, 1361016598, 1361016596, 1361016594, 1361016220, 1361016219, 1361016218, 1361016216, 1361016213, 1361016210, 1361016167, 1361016166, 1361016164, 1361016163, 1361016162, 1361016161, 1361016072, 1361016071, 1361016070, 1361016069, 1361016068, 1361016065, 1361016007, 1361016006, 1361016005, 1361016004, 1361016003, 1361016001, 1361015847, 1361015846, 1361015845, 1361015844, 1361015843, 1361015841, 1361015699, 1361015698, 1361015697, 1361015696, 1361015694, 1361015691, 1361015619, 1361015618, 1361015616, 1361015614, 1361015612, 1361015609, 1361015554, 1361015553, 1361015552, 1361015551, 1361015550, 1361015548, 1361015516, 1361015515, 1361015514, 1361015512, 1361015510, 1361015508, 1361015402, 1361015401, 1361015400, 1361015399, 1361015397, 1361015395, 1361015042, 1361015041, 1361015039, 1361015037, 1361015035, 1361015033, 1361014912, 1361014911, 1361014909, 1361014907, 1361014905, 1361014904, 1361014858, 1361014857, 1361014856, 1361014855, 1361014854, 1361014852, 1361014789, 1361014788, 1361014786, 1361014784, 1361014782, 1361014780, 1361014694, 1361014693, 1361014691, 1361014689, 1361014687, 1361014685, 1361014437, 1361014436, 1361014435, 1361014433, 1361014432, 1361014430, 1361014325, 1361014323, 1361014321, 1361014319, 1361014317, 1361014315, 1361014214, 1361014212, 1361014211, 1361014210, 1361014209, 1361014206, 1361014013, 1361014012, 1361014011, 1361014010, 1361014007, 1361014004, 1361013851, 1361013850, 1361013849, 1361013848, 1361013846, 1361013843, 1361013738, 1361013737, 1361013736, 1361013735, 1361013733, 1361013732, 1361012396, 1361012392, 1361012391, 1361012389, 1361012387, 1361012385, 1361011805, 1361011779, 1361011753, 1361011725, 1361011699, 1361011669, 1361011121, 1361011120, 1361011119, 1361011118, 1361011116, 1361011114, 1361011103, 1361011102, 1361011101, 1361011100, 1361011098, 1361011095, 1361011072, 1361011045, 1361011032, 1361011031, 1361011018, 1361010988, 1361006409, 1361006383, 1361006357, 1361006354, 1361006351, 1361006349, 1361003459, 1361003458, 1361003457, 1361003456, 1361003454, 1361003452, 1361003110, 1361003084, 1361003057, 1361003031, 1361003004, 1361002977, 1361002622, 1361002591, 1361002563, 1361002534, 1361002507, 1361002480, 1361002129, 1361002128, 1361002127, 1361002126, 1361002125, 1361002121] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.09920430183410645 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 19.818846940994263 time spend to save output : 0.10717010498046875 total time spend for step 0 : 19.92601704597473 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.98user 4.20system 1:01.56elapsed 31%CPU (0avgtext+0avgdata 1804648maxresident)k 178688inputs+344776outputs (49major+587109minor)pagefaults 0swaps