python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1733623 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 14101>200 please execute split_portfolio.py -i 22636503 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2977398'] with mtr_portfolio_ids : ['22636503'] and first list_photo_ids : [] new path : /proc/1733623/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 35.40452766418457 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri May 30 06:11:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-30 06:11:07.392609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-30 06:11:07.422781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-30 06:11:07.423058: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-30 06:11:07.424993: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-30 06:11:07.426757: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-30 06:11:07.427076: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-30 06:11:07.429174: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-30 06:11:07.430216: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-30 06:11:07.434507: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-30 06:11:07.435671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-30 06:11:07.436085: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-30 06:11:07.463184: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-30 06:11:07.465468: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8328000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-30 06:11:07.465503: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-30 06:11:07.599651: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2cfb76b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-30 06:11:07.599700: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-30 06:11:07.601044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-30 06:11:07.601136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-30 06:11:07.601173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-30 06:11:07.601207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-30 06:11:07.601236: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-30 06:11:07.601269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-30 06:11:07.601302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-30 06:11:07.601335: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-30 06:11:07.603035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-30 06:11:07.603131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-30 06:11:07.604263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-30 06:11:07.604311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-30 06:11:07.604336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-30 06:11:07.605677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3267 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-30 06:11:15.128252: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.488951683044434 time used to load_weights : 0.8980531692504883 0it [00:00, ?it/s] 19it [00:00, 184.73it/s] 38it [00:00, 181.84it/s] 58it [00:00, 187.48it/s] 80it [00:00, 198.36it/s] 101it [00:00, 197.81it/s]temp/1748578229_1733623_1355703516_68d4323ff6eb928bf4a14e33fb87132d.jpg temp/1748578229_1733623_1355703503_c820f8fe29a89601ec5a09cde2411fc4.jpg temp/1748578229_1733623_1355703502_538c2c8e6c7982257f6e7b7831deab77.jpg temp/1748578229_1733623_1355703499_79e0bf811ae1b8cc977084c3033724a9.jpg temp/1748578229_1733623_1355703498_75a94734ab4cf7e8fc777e5de1b4ebd2.jpg temp/1748578229_1733623_1355703496_790b5bbde2f86c0b8202b83403681a9a.jpg temp/1748578229_1733623_1355703493_77cb3bab6889a972d02d8f2621ce95d3.jpg temp/1748578229_1733623_1355703466_ae9600c313c373922cdb5c72a19f8685.jpg temp/1748578229_1733623_1355703463_76fc35234b67fd2ffe7bfd9d6334dba4.jpg temp/1748578229_1733623_1355703461_6d6e33222c14064e4c8428ef2d265527.jpg temp/1748578229_1733623_1355703460_0592d5a7cf3b0eff4660e64f45319a92.jpg temp/1748578229_1733623_1355703458_e92af5d6024b0294dc6758e61503e023.jpg temp/1748578229_1733623_1355703455_e1b2b32bcc1fafbcc7099ff1673f1ec0.jpg temp/1748578229_1733623_1355703379_716b5f636d3776520051f4ad9aba49c8.jpg temp/1748578229_1733623_1355703377_8f13ecfbfa7cd4401163fed9889a2520.jpg temp/1748578229_1733623_1355703374_8b387c8a5d8723197412d1ce54029d90.jpg temp/1748578229_1733623_1355703371_5a88dbd373e0f53cd38f0c28d19e5c11.jpg temp/1748578229_1733623_1355703368_0177a2e24c66a642fb8478febf66484d.jpg temp/1748578229_1733623_1355703364_b292f40981db1ab18f84d8f4d2c394ef.jpg temp/1748578229_1733623_1355703329_5275e54f6674bd6bba83c532f3154231.jpg temp/1748578229_1733623_1355703316_970d39a19cd12c7036621653e9facb02.jpg temp/1748578229_1733623_1355703308_80c751fed121610325770183e47aefec.jpg temp/1748578229_1733623_1355703295_0f74422765566b7c0afb1b69de677bb3.jpg temp/1748578229_1733623_1355703279_862fe30914ebf922cb6bcf44de471573.jpg temp/1748578229_1733623_1355703269_cd16f55b340bc7c04ed8f1264080da6e.jpg temp/1748578229_1733623_1355703253_bdc803c77c9da49beed780d986c3ab04.jpg temp/1748578229_1733623_1355703249_2d9e27193207016e4b59762986a30b77.jpg temp/1748578229_1733623_1355703245_1e626a56ed7485a894cd3ae551df49f0.jpg temp/1748578229_1733623_1355703243_52685715e9d4d34ad8eb53f0083578a4.jpg temp/1748578229_1733623_1355703241_1a63e862ef036cba8467330c9cd07308.jpg temp/1748578229_1733623_1355703240_48eb7a701e7764f65f07e6513bd7d5c8.jpg temp/1748578229_1733623_1355703226_7b41407442d828a614ab5824e5163207.jpg temp/1748578229_1733623_1355703223_8c0b5678eab2d8ac081868926b3f4290.jpg temp/1748578229_1733623_1355703219_cd7917b1440982a9cc1ba0604b54ab46.jpg temp/1748578229_1733623_1355703217_d0a76b8c44dba351f8487cdbc6ce6188.jpg temp/1748578229_1733623_1355703215_eaaad1551af1cee644490c8693523bbe.jpg temp/1748578229_1733623_1355703214_dc2bd69d91cdd7aabb777b0d520e3211.jpg temp/1748578229_1733623_1355703197_0a995b93e4e931ac0a67a74bc5a703ba.jpg temp/1748578229_1733623_1355703194_2f29be3f001696a06790883b0be22cea.jpg temp/1748578229_1733623_1355703191_42a1e310797fd2b3e10e116d63f249e9.jpg temp/1748578229_1733623_1355703189_43454032fefab5e01f4a989cf3113731.jpg temp/1748578229_1733623_1355703186_ba280f6bffa4667d45acf213f537dc80.jpg temp/1748578229_1733623_1355703184_e5c0461d5a1d92d25ac12e6f745083bc.jpg temp/1748578229_1733623_1355703169_a44f68a36f37a40d33419000848a9be1.jpg temp/1748578229_1733623_1355703165_aef5d380fa399e11b5e10b1b7cae9b15.jpg temp/1748578229_1733623_1355703161_7fe759161c915e836d6326da81cd2a1d.jpg temp/1748578229_1733623_1355703159_5d52cd7efa5e955e1723ef9b571e686a.jpg temp/1748578229_1733623_1355703157_22bd82ec18e2dad4306fca142c2c3a39.jpg temp/1748578229_1733623_1355703156_fa7ad2db92049bc244db0347705c79f6.jpg temp/1748578229_1733623_1355703145_b4e0bd79dc0980428cb5b36e6b6f3a9c.jpg temp/1748578229_1733623_1355703142_08b8e7935f3e97857e9dee9b953de27f.jpg temp/1748578229_1733623_1355703139_1b40d84568607eb74e8df47c9df43003.jpg temp/1748578229_1733623_1355703137_d7dd9950df7e3275224807a37a945ca0.jpg temp/1748578229_1733623_1355703135_8b4d2e23d703642646a7d20ca3b491b9.jpg temp/1748578229_1733623_1355703126_07600e2127706b3f645624cd641010bf.jpg temp/1748578229_1733623_1355703087_e66dc825d02575fc1a7aeef2c353ab5a.jpg temp/1748578229_1733623_1355703084_e2414a4ceaba9267a6b1acb52ffc0e16.jpg temp/1748578229_1733623_1355703076_869fa3911411b2df14addc4d6a6c3d43.jpg temp/1748578229_1733623_1355703074_90c39f2d132d16fbee36b399601b7040.jpg temp/1748578229_1733623_1355703072_a469f8fa836915dff8001fb38f52779c.jpg temp/1748578229_1733623_1355703061_8214a4b21c813132743060a4115934b2.jpg temp/1748578229_1733623_1355703035_fb86cad0f3030cfa50ddeaa200593bdf.jpg temp/1748578229_1733623_1355703031_863abb676348f798dc65661b39a4ade6.jpg temp/1748578229_1733623_1355703027_8e07965eda7b28e87d67f3246ac67ac8.jpg temp/1748578229_1733623_1355703024_44e3779c293c19d5992784ea9ad34609.jpg temp/1748578229_1733623_1355703022_195d9e1df9441c91cc6cb5a7774e490e.jpg temp/1748578229_1733623_1355703020_2c4fd6b627f422a66b6cc8cb30b65fca.jpg temp/1748578229_1733623_1355703017_2ac15360fb5b582f8ecc49a9865c51a4.jpg temp/1748578229_1733623_1355703014_6735b07029d78b8fba254ecff54b107e.jpg temp/1748578229_1733623_1355703010_c29a97236475e49e72db0f7d182d742b.jpg temp/1748578229_1733623_1355703007_3214dab91140f035ce03e326569c4f87.jpg temp/1748578229_1733623_1355702999_64e1bd9433cb080ce03a7e19c0c5db2b.jpg temp/1748578229_1733623_1355702997_f9aa91e68aa9ec94fbdb932800cebc0c.jpg temp/1748578229_1733623_1355702972_6732aef90e2a2931ab11182f0f88528c.jpg temp/1748578229_1733623_1355702967_17e4e976f69087e8545f29420fd56412.jpg temp/1748578229_1733623_1355702963_662c910aa2a3737fdcd8bab8d4e9a2c7.jpg temp/1748578229_1733623_1355702962_aed03bc6606a70acd81ce827a4c698db.jpg temp/1748578229_1733623_1355702960_bf08c71859f22117966f1e066ec81c92.jpg temp/1748578229_1733623_1355702958_a1f9e8eca7afacdd6a61cac4bd320b94.jpg temp/1748578229_1733623_1355702950_87121a7f69be15a695d48e44591b1553.jpg temp/1748578229_1733623_1355702947_3c4cf7ad96c46b5442e0b3df21aa97a0.jpg temp/1748578229_1733623_1355702943_c20a57c6cfd253d927254899f45c8c80.jpg temp/1748578229_1733623_1355702939_9f148718832c40403221ed9ae790dcb0.jpg temp/1748578229_1733623_1355702937_fbf8ed2763c6fae2648f6442af4fb03f.jpg temp/1748578229_1733623_1355702935_99aa735aa1bd075e16817a4e92ad95f1.jpg temp/1748578229_1733623_1355702910_c3db7afc5bc66531474d1244d0bbea28.jpg temp/1748578229_1733623_1355702907_9a2f04f4ae080045a71ab19bf32b8c47.jpg temp/1748578229_1733623_1355702905_f0f9a9db7faf456e07447ab9ba5574ba.jpg temp/1748578229_1733623_1355702904_5caaac0dcc99026dde71f9e472bcdf02.jpg temp/1748578229_1733623_1355702903_6815192daf9949cac22c76494905e55a.jpg temp/1748578229_1733623_1355702901_ddca78a5f808dd37d828a7544bb7ce7e.jpg temp/1748578229_1733623_1355702878_ec2cdd1de57e173781dde6dc971c6cab.jpg temp/1748578229_1733623_1355702874_7acd17c690ec965e0d2b7f25fed97027.jpg temp/1748578229_1733623_1355702870_337765e88252377e6cefc49f339ffb97.jpg temp/1748578229_1733623_1355702866_acc34dd3a1cfd1412055a4b5b8b5484d.jpg temp/1748578229_1733623_1355702864_5a705e25215bbf7f682e239613455588.jpg temp/1748578229_1733623_1355702862_e483e652a1d7ad59a35b566d74f193f3.jpg temp/1748578229_1733623_1355702852_a0e883c86d4a45787d669132c541b730.jpg temp/1748578229_1733623_1355702849_30bb7353e08111d103399fb0160e3ebb.jpg temp/1748578229_1733623_1355702846_327e7c91f9abb15741963d98ea204623.jpg temp/1748578229_1733623_1355702842_75b71ca9b34808482e29e69a3528efa9.jpg temp/1748578229_1733623_1355702840_3ad42918abfa2a110de93ca9d4b56e5a.jpg temp/1748578229_1733623_1355702838_2d62d0706bc865c7b914794a2380f691.jpg temp/1748578229_1733623_1355702829_635b9c2f54d657f9c4fada4fdd718f54.jpg temp/1748578229_1733623_1355702826_2c5c610d3ac64597c79a92b17b5205aa.jpg temp/1748578229_1733623_1355702823_d78b977eb0797de9668c01f40fb04db5.jpg temp/1748578229_1733623_1355702819_0c0f8b4972fa6672c6b6f21879e77a4c.jpg temp/1748578229_1733623_1355702815_155ab35575a63c96b1f484aaf051fdac.jpg temp/1748578229_1733623_1355702813_7c4713a9b5dcce6a5c57eeaef77753f7.jpg temp/1748578229_1733623_1355702800_9195c526fb107d32f6adb723ae08c1cc.jpg temp/1748578229_1733623_1355702797_639fee3fd17d7b5ee225e50c818e18fe.jpg temp/1748578229_1733623_1355702794_668033503128b604d7edd38d359f55b8.jpg temp/1748578229_1733623_1355702793_2bf2b5a31ef32c4c068323635eaa0d18.jpg 121it [00:00, 193.03it/s] 142it [00:00, 197.01it/s] 163it [00:00, 196.58it/s] 183it [00:00, 193.21it/s] 200it [00:01, 192.88it/s]2025-05-30 06:11:21.196775: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-30 06:11:21.428694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-30 06:11:21.898109: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-30 06:11:21.906997: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748578229_1733623_1355702790_514369ad5f73ee284b495e61b7165d75.jpg temp/1748578229_1733623_1355702786_e308e5e37cd84eb37aa200214c1f27d6.jpg temp/1748578229_1733623_1355702773_81c6bae8012e8db51b0707bb60d25e17.jpg temp/1748578229_1733623_1355702770_c595bba9d8b84eedfbf5595748f8e4ab.jpg temp/1748578229_1733623_1355702767_ad8b0a6924cd2232beb1c50a0a9e3d78.jpg temp/1748578229_1733623_1355702764_a842a99cfec3af815854570771d20fa1.jpg temp/1748578229_1733623_1355702761_cf2e668ffac9bc827e7f51a65c6a5612.jpg temp/1748578229_1733623_1355702758_1a36a6269fccd601fb04f62576337de0.jpg temp/1748578229_1733623_1355702748_61fcc37baeab7205b87b328dd9e257a7.jpg temp/1748578229_1733623_1355702746_13675fa3527515ee54e8f24ae9948d3f.jpg temp/1748578229_1733623_1355702744_22505c03117f059f2e8e815d1d1a63f6.jpg temp/1748578229_1733623_1355702742_6a0586bf982a2dc054c51942a89c0676.jpg temp/1748578229_1733623_1355702738_9343219c67a1550b2c4f8de8670efc21.jpg temp/1748578229_1733623_1355702735_80c5b0049176843b0610c0392c876a21.jpg temp/1748578229_1733623_1355702469_1239ff19525b9932d71cd452131aee7b.jpg temp/1748578229_1733623_1355702440_f971bfa8d1e40005a87c4190f121c87e.jpg temp/1748578229_1733623_1355702435_d8330059bb311b9e040489fe66ddabfa.jpg temp/1748578229_1733623_1355702433_7dd56ff53ef9f6081ac8d46713f14bec.jpg temp/1748578229_1733623_1355702430_fd4e2d1166fb4b230d709ed02b687865.jpg temp/1748578229_1733623_1355702427_35b4401e41e285d420ff780c268ce370.jpg temp/1748578229_1733623_1355701380_6f722c2fbd9a784a2f5ee654104d163a.jpg temp/1748578229_1733623_1355701343_1abeec238a5662eccca3fa32f37d577b.jpg temp/1748578229_1733623_1355701317_3c4fb4c24c306ba58e7e536f363a3086.jpg temp/1748578229_1733623_1355701292_d8b5761864b01f8c1b2f4d6cb8a04d15.jpg temp/1748578229_1733623_1355701289_884a8a0754b781e7712f6ec17d0a0a23.jpg temp/1748578229_1733623_1355701287_b3be01f4e2b771b10f785d5ae63343f7.jpg temp/1748578229_1733623_1355701272_16d508b4e4909d8786d870551ee01e66.jpg temp/1748578229_1733623_1355701269_dd0d095eeb45ac8d5a22077ad01fb0b0.jpg temp/1748578229_1733623_1355701266_f0d1051ac061c698c0fbdbd75e7a602b.jpg temp/1748578229_1733623_1355701264_3c36ba5367866e14ac71fb3492128aab.jpg temp/1748578229_1733623_1355701261_2ba5aa73edb316cd887de0e5dad9539b.jpg temp/1748578229_1733623_1355701259_41e5b48785a3458ca5003cf21de648ec.jpg temp/1748578229_1733623_1355701249_9e171d88f85e20d33f647a9120f8db01.jpg temp/1748578229_1733623_1355701246_9ee1cd74dc35d79eed4e3cbf0a309546.jpg temp/1748578229_1733623_1355701235_2b971877608d7b372ed7d72a440e440a.jpg temp/1748578229_1733623_1355701207_f070a303a5cf0cc1b255cd7166a21ea3.jpg temp/1748578229_1733623_1355701181_479d26ce0026363d2bb36f2e6c6998e8.jpg temp/1748578229_1733623_1355701141_3793ff2747632286dc36ba113eebc1c9.jpg temp/1748578229_1733623_1355700572_c85bd8287c7c9bc9d7ed85922973b30e.jpg temp/1748578229_1733623_1355700519_b28b332a3e07a87adda9de37fa8446ac.jpg temp/1748578229_1733623_1355700459_5d42c997d364a0f9fef197c4d06b3d8a.jpg temp/1748578229_1733623_1355700382_50cdb3a2cb97fa00cb1f7cb50f4e3f07.jpg temp/1748578229_1733623_1355700304_62fabfef256749091ef349d471b3736c.jpg temp/1748578229_1733623_1355700213_02f80869fd5daf0c73addd4ea4bc685a.jpg temp/1748578229_1733623_1355699628_84f15ea6d86042ee481095e489cabc22.jpg temp/1748578229_1733623_1355699548_7e01a7423a6994f826bd4c52ca23b55a.jpg temp/1748578229_1733623_1355699342_f2380d5ae77b36a051595d3b286844e0.jpg temp/1748578229_1733623_1355699261_f77a829642ffd2755fe1a5b96b5d0090.jpg temp/1748578229_1733623_1355699146_710b0edc4445718c945846e8be98dd6d.jpg temp/1748578229_1733623_1355699070_b466e61e5ca708bca65ce150f757a95b.jpg temp/1748578229_1733623_1355698743_b335c83a7ddfb885c6703f2b25fb1d76.jpg temp/1748578229_1733623_1355698691_3ca38f8bb99823641bcf7652ecc3542b.jpg temp/1748578229_1733623_1355698624_b06004c11d6e837eba32d50cb258c726.jpg temp/1748578229_1733623_1355698586_2a35ff000303ba46a9970fc04d7b0a75.jpg temp/1748578229_1733623_1355698450_b010b464a703e22c7ff077f08b3b89bd.jpg temp/1748578229_1733623_1355698412_c9b8fcae6c46bf3bddd7613252fc3816.jpg temp/1748578229_1733623_1355698338_33a2e7fdefab728a76c5dab473521da4.jpg temp/1748578229_1733623_1355698310_04cacaa72421004d20a80b0e91b1f16f.jpg temp/1748578229_1733623_1355698259_0e5f107511f7d3f7a15575eb2263335a.jpg temp/1748578229_1733623_1355698226_53841ab6a4383abc37be45cb807188ab.jpg temp/1748578229_1733623_1355698197_99e9c7e806bd8111788f4ba23cdd2426.jpg temp/1748578229_1733623_1355698153_13c2a0e503d8858132265b90e45f640a.jpg temp/1748578229_1733623_1355697183_093eb49b5e381f0f91660bc8f108211a.jpg temp/1748578229_1733623_1355697109_cee28eca406d6a483ff73c68bf280a90.jpg temp/1748578229_1733623_1355697057_9a387e59804f9d8b7880390bf833f30a.jpg temp/1748578229_1733623_1355697006_b306a47d6a9e40dd82761a434bd17454.jpg temp/1748578229_1733623_1355696907_fda781f2779a36d17e9b1a70af3dc2e9.jpg temp/1748578229_1733623_1355696877_4eae4c392ae3d97cf7560589ec299cae.jpg temp/1748578229_1733623_1355696465_3acbdcdb2a7aecb83c1f63b51742ba14.jpg temp/1748578229_1733623_1355696461_af99aaf4ca470e5b005fbbe39733f230.jpg temp/1748578229_1733623_1355696457_3c8c0906bb4d27533101f73f0b41af5f.jpg temp/1748578229_1733623_1355696455_fbe9b233f105e42b998e61cf67824946.jpg temp/1748578229_1733623_1355696454_89ea56235812d88d0b76f89f6019f057.jpg temp/1748578229_1733623_1355696452_bfc3523a25f3db643084e51dce24f700.jpg temp/1748578229_1733623_1355696436_5d9280d0f4720dc733c0a886df55499d.jpg temp/1748578229_1733623_1355696434_33c93ffaccab2c3768737cd9fa528a16.jpg temp/1748578229_1733623_1355696432_a7d948630aa81c0497bd76f05fdfd5dc.jpg temp/1748578229_1733623_1355696430_a99dfa09f9697b13e42d10d1a2c0e059.jpg temp/1748578229_1733623_1355696428_764831a8a1edd7a2e05ca72af39c2ec9.jpg temp/1748578229_1733623_1355696424_f416e366f0d55e2f2a7a7e2f27992543.jpg temp/1748578229_1733623_1355696411_8350ba60b58c65cce6b9062cc55a3798.jpg temp/1748578229_1733623_1355696409_310159bb36ad4c356c56145ecac0a503.jpg temp/1748578229_1733623_1355696404_a93faae2274c3b2e0f05af7efef4decf.jpg temp/1748578229_1733623_1355696402_d08695754a4b2948a43ddcc2e85c98e6.jpg temp/1748578229_1733623_1355696399_6c4e9a3695bc8c88a75547bf71a90c87.jpg temp/1748578229_1733623_1355696397_ddff72381a8bc39b4f7445fd736be6e7.jpg temp/1748578229_1733623_1355696383_7e773e8fbc79990688344feb78de170a.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1355703516, 1355703503, 1355703502, 1355703499, 1355703498, 1355703496, 1355703493, 1355703466, 1355703463, 1355703461, 1355703460, 1355703458, 1355703455, 1355703379, 1355703377, 1355703374, 1355703371, 1355703368, 1355703364, 1355703329, 1355703316, 1355703308, 1355703295, 1355703279, 1355703269, 1355703253, 1355703249, 1355703245, 1355703243, 1355703241, 1355703240, 1355703226, 1355703223, 1355703219, 1355703217, 1355703215, 1355703214, 1355703197, 1355703194, 1355703191, 1355703189, 1355703186, 1355703184, 1355703169, 1355703165, 1355703161, 1355703159, 1355703157, 1355703156, 1355703145, 1355703142, 1355703139, 1355703137, 1355703135, 1355703126, 1355703087, 1355703084, 1355703076, 1355703074, 1355703072, 1355703061, 1355703035, 1355703031, 1355703027, 1355703024, 1355703022, 1355703020, 1355703017, 1355703014, 1355703010, 1355703007, 1355702999, 1355702997, 1355702972, 1355702967, 1355702963, 1355702962, 1355702960, 1355702958, 1355702950, 1355702947, 1355702943, 1355702939, 1355702937, 1355702935, 1355702910, 1355702907, 1355702905, 1355702904, 1355702903, 1355702901, 1355702878, 1355702874, 1355702870, 1355702866, 1355702864, 1355702862, 1355702852, 1355702849, 1355702846, 1355702842, 1355702840, 1355702838, 1355702829, 1355702826, 1355702823, 1355702819, 1355702815, 1355702813, 1355702800, 1355702797, 1355702794, 1355702793, 1355702790, 1355702786, 1355702773, 1355702770, 1355702767, 1355702764, 1355702761, 1355702758, 1355702748, 1355702746, 1355702744, 1355702742, 1355702738, 1355702735, 1355702469, 1355702440, 1355702435, 1355702433, 1355702430, 1355702427, 1355701380, 1355701343, 1355701317, 1355701292, 1355701289, 1355701287, 1355701272, 1355701269, 1355701266, 1355701264, 1355701261, 1355701259, 1355701249, 1355701246, 1355701235, 1355701207, 1355701181, 1355701141, 1355700572, 1355700519, 1355700459, 1355700382, 1355700304, 1355700213, 1355699628, 1355699548, 1355699342, 1355699261, 1355699146, 1355699070, 1355698743, 1355698691, 1355698624, 1355698586, 1355698450, 1355698412, 1355698338, 1355698310, 1355698259, 1355698226, 1355698197, 1355698153, 1355697183, 1355697109, 1355697057, 1355697006, 1355696907, 1355696877, 1355696465, 1355696461, 1355696457, 1355696455, 1355696454, 1355696452, 1355696436, 1355696434, 1355696432, 1355696430, 1355696428, 1355696424, 1355696411, 1355696409, 1355696404, 1355696402, 1355696399, 1355696397, 1355696383] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.5284032821655273 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 17.391595602035522 time spend to save output : 0.5544602870941162 total time spend for step 0 : 17.94605588912964 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 15.93user 4.81system 0:59.82elapsed 34%CPU (0avgtext+0avgdata 2175128maxresident)k 545248inputs+347872outputs (268major+771530minor)pagefaults 0swaps