python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3624400 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2700>100 please execute split_portfolio.py -i 21247140 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2656062'] with mtr_portfolio_ids : ['21247140'] and first list_photo_ids : [] new path : /proc/3624400/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 13.293368577957153 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Tue Mar 11 07:52:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-03-11 07:52:51.396944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-11 07:52:51.424676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-11 07:52:51.425066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-11 07:52:51.427665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-11 07:52:51.430072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-11 07:52:51.430519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-11 07:52:51.433083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-11 07:52:51.434390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-11 07:52:51.439657: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-11 07:52:51.440898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-11 07:52:51.441470: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-11 07:52:51.467205: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-11 07:52:51.469615: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1dec000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-11 07:52:51.469661: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-11 07:52:51.695397: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x17999200 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-11 07:52:51.695463: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-11 07:52:51.696490: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-11 07:52:51.696649: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-11 07:52:51.696673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-11 07:52:51.696692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-11 07:52:51.696708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-11 07:52:51.696724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-11 07:52:51.696741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-11 07:52:51.696758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-11 07:52:51.697693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-11 07:52:51.697846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-11 07:52:51.698507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-11 07:52:51.698537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-11 07:52:51.698549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-11 07:52:51.699977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2944 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-03-11 07:52:58.516741: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-11 07:52:58.517345: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.262048482894897 time used to load_weights : 0.2318260669708252 0it [00:00, ?it/s] 100it [00:00, 1683.80it/s]2025-03-11 07:53:02.221000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-11 07:53:02.424134: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-11 07:53:02.702534: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-03-11 07:53:02.713531: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1741675950_3624400_1342863907_f9d97a22a006786258c6a8c41876fcd8.jpg temp/1741675950_3624400_1342863900_9883a4ba029768284e0f752bd0850a75.jpg temp/1741675950_3624400_1342863897_686ffa47e9785b4319ae25c14473a51a.jpg temp/1741675950_3624400_1342863892_4dafaac267c6222b9e98aae38a981309.jpg temp/1741675950_3624400_1342863887_d3ff18ea1ee41833472b0b06c166f6bb.jpg temp/1741675950_3624400_1342863863_bb2cb96278e811472dc4a80ed3e466d7.jpg temp/1741675950_3624400_1342863580_46adf7679ee3c150bbc57f24704a2681.jpg temp/1741675950_3624400_1342863566_f80d57b76e9826e6b62612edfaeab2ae.jpg temp/1741675950_3624400_1342863560_4757f4070a303388f2c4f7603d8c5dec.jpg temp/1741675950_3624400_1342863535_1916a14dba0569ca42ad76e01be3f4e2.jpg temp/1741675950_3624400_1342863508_ec8aac02991f7d8d0b1e1ca0903d0106.jpg temp/1741675950_3624400_1342863483_ceabedc2594c7015f17fb46cac825004.jpg temp/1741675950_3624400_1342863231_eac3b4c4ee21f06bc451f7db7511eca7.jpg temp/1741675950_3624400_1342863186_9e4be375f568e041a38f0b84c3baa965.jpg temp/1741675950_3624400_1342863177_048a1c59b93dcbb78b56be5a1e823c60.jpg temp/1741675950_3624400_1342863140_7af4f2e8eb042fb51c0c1bbb1649ce3c.jpg temp/1741675950_3624400_1342863125_6d24061057501c1c389299451625c779.jpg temp/1741675950_3624400_1342863118_e07d5640d3749a12af929e597ded554c.jpg temp/1741675950_3624400_1342862815_1b4396639b10cc06925938e1920ca516.jpg temp/1741675950_3624400_1342862806_92946b7ca675b84cfb50b819afe0b089.jpg temp/1741675950_3624400_1342862795_7336facf76bbdd522a78ff04bd9cfcb6.jpg temp/1741675950_3624400_1342862781_1b0cbfc3dd922dcd967b9c9678690be8.jpg temp/1741675950_3624400_1342862774_e5b543e042684c861ba586265acf3611.jpg temp/1741675950_3624400_1342862746_2f5f9f87d9b45c86f538a2934837d560.jpg temp/1741675950_3624400_1342862684_1b3d76cd002efd3b310db7291602b772.jpg temp/1741675950_3624400_1342862679_eb256b87667c4850615b8be66fa5c4dd.jpg temp/1741675950_3624400_1342862673_bb87298a400bd9c537cdc9c1b7334a5c.jpg temp/1741675950_3624400_1342862640_50d520e295c609f25a293cacf9a4ce79.jpg temp/1741675950_3624400_1342862633_0ce2bc308d09826fe237768dce8ee86f.jpg temp/1741675950_3624400_1342862525_d510648ea5a4def5d17ab2bcb9290bdc.jpg temp/1741675950_3624400_1342862467_e7f683cd9968d4bb53b5c942cc0ff076.jpg temp/1741675950_3624400_1342862463_fb9230b4051890402f3f45bdb49d5501.jpg temp/1741675950_3624400_1342862451_012dd29aae4abf641232149a03df01d0.jpg temp/1741675950_3624400_1342862446_e4ef65b954c37bfa0f61a7640be29b2f.jpg temp/1741675950_3624400_1342862441_b31ca70d03a35d84fd57ef940f747e61.jpg temp/1741675950_3624400_1342862437_563cf328d9bd44e413c402bd9bfacc6a.jpg temp/1741675950_3624400_1342862364_15dfc4d2d786a8b56c261ff3fb9ab47f.jpg temp/1741675950_3624400_1342862352_7a858967c86de291b153b1d42822e311.jpg temp/1741675950_3624400_1342862330_39c1cc4c1bc2f8043b504d79cfff85d4.jpg temp/1741675950_3624400_1342862299_66bf80a7f0af44dd1819bb3871931c5b.jpg temp/1741675950_3624400_1342862288_303e620f8d578ed2d1290623d35cae3b.jpg temp/1741675950_3624400_1342862262_16446eab2c1a59ea864758e04b5dcd81.jpg temp/1741675950_3624400_1342848852_efc1322a3663d887ef7285647c7c6d3d.jpg temp/1741675950_3624400_1342848829_ee9c26369f37b2a09e91a14d859fdc2b.jpg temp/1741675950_3624400_1342848808_e9a5fb49075f6b8e5fbeafd4f27d135f.jpg temp/1741675950_3624400_1342848803_9bf0dffd7fc9e83a90a466cd140d4b5f.jpg temp/1741675950_3624400_1342848797_717157f09f65981161b302909a15d2ef.jpg temp/1741675950_3624400_1342848783_4c4741cbd4d3aecd7201f73b07a0df9e.jpg temp/1741675950_3624400_1342848744_3974506a18c0a2be734378ec2da74e83.jpg temp/1741675950_3624400_1342848728_5cf4b7c1dd6d4901dbd190b1c8e70361.jpg temp/1741675950_3624400_1342848715_0fe835fd4077613cdeeb073997e40cda.jpg temp/1741675950_3624400_1342848708_0b9d965b2bb725e7a74c9c303ac1917a.jpg temp/1741675950_3624400_1342848701_db7283cfac259b522d10fa5ff0eea2d5.jpg temp/1741675950_3624400_1342848696_ba26fa3f0eb0328739089e695ddc03cb.jpg temp/1741675950_3624400_1342848663_a6ecdbad68ea04062ba2f835d3c70c2c.jpg temp/1741675950_3624400_1342848654_963996df0bb91cfe812e9c91993cce9e.jpg temp/1741675950_3624400_1342848645_af100b19edd85f13adc00ebd09b363c8.jpg temp/1741675950_3624400_1342848640_5071ffa146a5e3cca2c3f88cb8769f1b.jpg temp/1741675950_3624400_1342848635_f42a522b728f27c9dbb2d32dee796db7.jpg temp/1741675950_3624400_1342848630_3b8438aaecb012de2f1f1f61341d03d7.jpg temp/1741675950_3624400_1342848613_b0fa37e52a11b3a6e0d2c7eac05463c7.jpg temp/1741675950_3624400_1342848610_9f287a45cf8079dc2db6649b7c1301c8.jpg temp/1741675950_3624400_1342848607_73b2aa9348a8c622809b121533c91a33.jpg temp/1741675950_3624400_1342848599_c2debd67dd6cbdb434954d5499cdd393.jpg temp/1741675950_3624400_1342848588_77f8c5d5fe2fa341c0fae252cf1d356b.jpg temp/1741675950_3624400_1342848580_735171aefb95a56230dea9214ced5e2f.jpg temp/1741675950_3624400_1342848557_68f5a87ce5afce7e71212e1b6870a826.jpg temp/1741675950_3624400_1342848553_271f1c49579f6ccec2db31622a813994.jpg temp/1741675950_3624400_1342848548_741315069bd39dc567571b36973e1d86.jpg temp/1741675950_3624400_1342848543_544447da0cd2d51845086f235eb22660.jpg temp/1741675950_3624400_1342848539_8bf56f59ad44cd6eccc748fc302f90ae.jpg temp/1741675950_3624400_1342848534_a9933c1656f9115441c0c735b0e967a0.jpg temp/1741675950_3624400_1342848448_d14bd807871971b84d06f75a781a84b3.jpg temp/1741675950_3624400_1342848417_fd2972881b09b6f66ce94aad9958ed81.jpg temp/1741675950_3624400_1342848368_0c3469c296cb99d8c1ca11972ebe2903.jpg temp/1741675950_3624400_1342848314_a2f7333bab17957a90b291f9a783299f.jpg temp/1741675950_3624400_1342848264_c0a3492ebb3f08eb2f51c7bf83c1bf8c.jpg temp/1741675950_3624400_1342848218_da0de6f741c77d5ab294ce9dd0cc2d68.jpg temp/1741675950_3624400_1342847976_49ecca43ac3086b6586839f1e93c1237.jpg temp/1741675950_3624400_1342847972_5a60acbadac3c6ad6710d2cabce98731.jpg temp/1741675950_3624400_1342847946_9fc9717e70a81cbff7289b03562b55a2.jpg temp/1741675950_3624400_1342847920_29897b4e8d55b48873c5ccbb793fef1e.jpg temp/1741675950_3624400_1342847891_0d5276a18fded7d65a848f02e6f4ddf4.jpg temp/1741675950_3624400_1342847871_591cf49e57914e456237271eff9c229b.jpg temp/1741675950_3624400_1342847814_0cf3245e5a0d48cefa9a91cff7fbf852.jpg temp/1741675950_3624400_1342847778_63d98189026fdffd4834d0f1998c3bd9.jpg temp/1741675950_3624400_1342847738_fb3d9c92003455ae25209e772328ba4a.jpg temp/1741675950_3624400_1342847699_97eb4859a34a569efcea31601197641e.jpg temp/1741675950_3624400_1342847670_889bd54849df3421422b27017ecbebd1.jpg temp/1741675950_3624400_1342847630_b7eeab3a261ed9c6032fc44c3bd8df32.jpg temp/1741675950_3624400_1342847336_310c9ce254bf29eacccd726a7373e9dd.jpg temp/1741675950_3624400_1342847256_29058b3886f7f313a36ad327cd99651f.jpg temp/1741675950_3624400_1342847230_28bf363d8039e05452739ba9f3bbcc24.jpg temp/1741675950_3624400_1342847196_f31ab3711b65895cefe95359a8dfa955.jpg temp/1741675950_3624400_1342847170_83cf441585c5dd0eda2c8045d84872f7.jpg temp/1741675950_3624400_1342847164_83b4291ca9b13bb9dcad2a03c140bb34.jpg temp/1741675950_3624400_1342847121_f9e3e4a4c704194ad5625160b20d1e77.jpg temp/1741675950_3624400_1342847115_ee076a6b9f0766e733cb23f9818e8e42.jpg temp/1741675950_3624400_1342847110_236e7a292abbeb2d571a53f9c5e1d508.jpg temp/1741675950_3624400_1342847098_55a46deaecf973106bff1610d299a521.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1342863907, 1342863900, 1342863897, 1342863892, 1342863887, 1342863863, 1342863580, 1342863566, 1342863560, 1342863535, 1342863508, 1342863483, 1342863231, 1342863186, 1342863177, 1342863140, 1342863125, 1342863118, 1342862815, 1342862806, 1342862795, 1342862781, 1342862774, 1342862746, 1342862684, 1342862679, 1342862673, 1342862640, 1342862633, 1342862525, 1342862467, 1342862463, 1342862451, 1342862446, 1342862441, 1342862437, 1342862364, 1342862352, 1342862330, 1342862299, 1342862288, 1342862262, 1342848852, 1342848829, 1342848808, 1342848803, 1342848797, 1342848783, 1342848744, 1342848728, 1342848715, 1342848708, 1342848701, 1342848696, 1342848663, 1342848654, 1342848645, 1342848640, 1342848635, 1342848630, 1342848613, 1342848610, 1342848607, 1342848599, 1342848588, 1342848580, 1342848557, 1342848553, 1342848548, 1342848543, 1342848539, 1342848534, 1342848448, 1342848417, 1342848368, 1342848314, 1342848264, 1342848218, 1342847976, 1342847972, 1342847946, 1342847920, 1342847891, 1342847871, 1342847814, 1342847778, 1342847738, 1342847699, 1342847670, 1342847630, 1342847336, 1342847256, 1342847230, 1342847196, 1342847170, 1342847164, 1342847121, 1342847115, 1342847110, 1342847098] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 1.8878366947174072 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 19.32664942741394 time spend to save output : 1.8947319984436035 total time spend for step 0 : 21.221381425857544 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.04user 4.71system 0:40.86elapsed 45%CPU (0avgtext+0avgdata 2088840maxresident)k 199312inputs+184928outputs (162major+664884minor)pagefaults 0swaps