python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 252643 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 1272>100 please execute split_portfolio.py -i 25480052 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3392477'] with mtr_portfolio_ids : ['25480052'] and first list_photo_ids : [] new path : /proc/252643/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 16.68871784210205 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Wed Jul 30 09:49:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-07-30 09:49:46.862971: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-07-30 09:49:46.895595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-30 09:49:46.895884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-30 09:49:46.898089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-30 09:49:46.900042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-30 09:49:46.900388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-30 09:49:46.902756: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-30 09:49:46.904039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-30 09:49:46.908168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-30 09:49:46.909198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-30 09:49:46.909517: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-07-30 09:49:46.935392: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493035000 Hz 2025-07-30 09:49:46.936975: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa960000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-07-30 09:49:46.937021: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-07-30 09:49:47.060257: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13a7a8d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-07-30 09:49:47.060303: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-07-30 09:49:47.061356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-30 09:49:47.061447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-30 09:49:47.061477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-30 09:49:47.061504: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-30 09:49:47.061530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-30 09:49:47.061556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-30 09:49:47.061582: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-30 09:49:47.061608: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-30 09:49:47.062960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-30 09:49:47.063024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-30 09:49:47.063884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-07-30 09:49:47.063910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-07-30 09:49:47.063924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-07-30 09:49:47.065321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3470 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.080549240112305 time used to load_weights : 0.1750495433807373 0it [00:00, ?it/s] 100it [00:00, 1414.18it/s]2025-07-30 09:49:56.234118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-30 09:49:56.461133: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-30 09:49:56.530787: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-07-30 09:49:56.533988: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1753861767_252643_1374421544_9d8dcb60a67a3d7917343268ccfe1d7a.jpg temp/1753861767_252643_1374421510_2d2b991ea75de61a96f8bd17143dd8f8.jpg temp/1753861767_252643_1374421474_04ce1c84a2a0894119b894b359c72804.jpg temp/1753861767_252643_1374421441_18ca68fd4c2319c9394b8e7e6ac5b07e.jpg temp/1753861767_252643_1374421407_a2daeda68d1293ebedf51834ef6246bf.jpg temp/1753861767_252643_1374421390_39825a32ee551fd843a4a4f3919eaeda.jpg temp/1753861767_252643_1374421359_5661f410ed92cb0a2f2b9f43aed7c7df.jpg temp/1753861767_252643_1374421357_3223369e839bd55ebf54c9e05724d240.jpg temp/1753861767_252643_1374421355_572053c73a62f775549d81dee0d58487.jpg temp/1753861767_252643_1374421352_3064fe24b50941d9ea82257e1923f9c9.jpg temp/1753861767_252643_1374421350_3cf5a21f77e37e9c0ce63c25b49c6a11.jpg temp/1753861767_252643_1374421259_907cdbd2bd367570f2ce6f3401227a5f.jpg temp/1753861767_252643_1374421251_46848d6938828e55504b3a5cf9378c4f.jpg temp/1753861767_252643_1374421217_973a553b83178f525a3dc635582c19fe.jpg temp/1753861767_252643_1374421213_228f26ebec1e160a839ac8c4b7ee4f6d.jpg temp/1753861767_252643_1374420883_0c568edc6974950ef06f52363fff4ae6.jpg temp/1753861767_252643_1374420882_ce436e637e3b6acd06d74059b64f6b82.jpg temp/1753861767_252643_1374420880_47597b8610fdd41fe8ed8c0cd5a4c673.jpg temp/1753861767_252643_1374420877_dc1d3f76154e685dcbda50cb316bca44.jpg temp/1753861767_252643_1374420875_d99a22f9181ffd719c6b1273ef371f48.jpg temp/1753861767_252643_1374420871_16106ea57cac2010469766499b418bde.jpg temp/1753861767_252643_1374420830_a3b501641b956084f09a484936dd0bca.jpg temp/1753861767_252643_1374420826_ce244e0d01e05e027f2de7e59f8c91e3.jpg temp/1753861767_252643_1374420823_ab7dda33f1e67737892fca97941bc798.jpg temp/1753861767_252643_1374420782_7a293806796cbe0b4cf102f378fa13ad.jpg temp/1753861767_252643_1374420781_de260426052378874165aaf75bed9af8.jpg temp/1753861767_252643_1374420754_f04155cb1c853bbd43ac3de2e9c89e29.jpg temp/1753861767_252643_1374420750_cfcd436090e19b0f2c5b6a8254083238.jpg temp/1753861767_252643_1374420747_399606fa91ccce5c2b6c81c9b52d1f09.jpg temp/1753861767_252643_1374420744_45ef2c5bc6e8f35000d81adc3d2b8521.jpg temp/1753861767_252643_1374420561_67c5fe5a686930c949a13a15181dcd4e.jpg temp/1753861767_252643_1374420556_e7f20021c66ea9a431e73653600cf70e.jpg temp/1753861767_252643_1374420344_2d82c2e58bdb9e67ba1cad16d7165f4f.jpg temp/1753861767_252643_1374420342_f35dc0aa0468a23ae680002606b3511f.jpg temp/1753861767_252643_1374420220_166528104390418d48049f3cdf06a146.jpg temp/1753861767_252643_1374420216_2ffaef632a951571b9849cf0ed3d0bc3.jpg temp/1753861767_252643_1374420212_91d0d16ee8b5dcb214881d7371e29966.jpg temp/1753861767_252643_1374420211_98a620e66a6f63268b83c30f67aaa1ae.jpg temp/1753861767_252643_1374420208_8e02664d1145e05b7a49552659afe0ac.jpg temp/1753861767_252643_1374420207_92a29a592fa72b2db1715c0f82e406cb.jpg temp/1753861767_252643_1374420205_7d2d8d8f3cd88ceeff5ac003d81a5bff.jpg temp/1753861767_252643_1374420177_ebc1aa0456fe56f233c98c7e2daf190c.jpg temp/1753861767_252643_1374419999_17f61384869524be54ee3f3ac0e206e9.jpg temp/1753861767_252643_1374419996_932fe24ff2839d1500c45f7674b7124e.jpg temp/1753861767_252643_1374419992_f8a39ae2b01a538f8fdaca5f356866e7.jpg temp/1753861767_252643_1374419990_002ed7d83139749d228fea9f12ddd250.jpg temp/1753861767_252643_1374419988_4e9beca3671e52e17c65a2745626dbf6.jpg temp/1753861767_252643_1374419983_d015fd2861bc01654053663bc8fd38f6.jpg temp/1753861767_252643_1374419959_8bbbea182505a16273f90e8e161c4225.jpg temp/1753861767_252643_1374419955_db861e7abeb858c8f17818ac0f49b99b.jpg temp/1753861767_252643_1374419935_1ed6a2ea1b36fe10d9b3cba4e4817dac.jpg temp/1753861767_252643_1374419898_af202f5b91c76ab4f2c795597f3ca416.jpg temp/1753861767_252643_1374419864_4e260e7c37949a1b670859a00531af92.jpg temp/1753861767_252643_1374419827_6abbf6e16f494a18713632cbfbfaa4e8.jpg temp/1753861767_252643_1374419235_d0b2912db4485be8efcbe49d9282bfcd.jpg temp/1753861767_252643_1374418681_c263824a5b1c32a0724cfa84550646e4.jpg temp/1753861767_252643_1374418312_24f02a85b0b23af0803c25e89b678f3e.jpg temp/1753861767_252643_1374417824_121382f2b926bf7ff8bbd38583f951b4.jpg temp/1753861767_252643_1374416810_38fa12a446cbc9dca9f31d8b364efea3.jpg temp/1753861767_252643_1374416724_83acfffa66e31fad335bb679032da9d7.jpg temp/1753861767_252643_1374416661_83ad992ce5662119760d736eb261eb8e.jpg temp/1753861767_252643_1374416576_a06eabab13bfc041aa5746129d2f14b2.jpg temp/1753861767_252643_1374416515_f3eea413630771eeb856c546b481d580.jpg temp/1753861767_252643_1374416429_a590e3ef757f3bb0e32f9e0ea823475a.jpg temp/1753861767_252643_1374415718_211312e85beec56b0c10d9c9851647bb.jpg temp/1753861767_252643_1374415715_924d7fe8ba342f191a44ee056978a24e.jpg temp/1753861767_252643_1374415713_1abfe71ab69cda9cf800ffd2410e596e.jpg temp/1753861767_252643_1374415708_195dc3c4ac57bc3f3c9287653f5516ae.jpg temp/1753861767_252643_1374415681_40dd2102b53bc729862fd769469edb4f.jpg temp/1753861767_252643_1374415679_018637309ba0f0331e47b00e9457eea0.jpg temp/1753861767_252643_1374415677_36b0401e87888d5733de02b746e6d3d8.jpg temp/1753861767_252643_1374415674_7e2fa12d099f0804d579b11a3d78fcab.jpg temp/1753861767_252643_1374415671_946c5d13a5b0b9f167bdfc0ab1e47585.jpg temp/1753861767_252643_1374415593_f5201d11991d9003dff030a2c9a6185f.jpg temp/1753861767_252643_1374415589_ed7cb17f308e80f0a0db51589bb88e41.jpg temp/1753861767_252643_1374415585_ef154a915e13eb63a39b0170ca382cab.jpg temp/1753861767_252643_1374415567_1997fab85768923072e1619f78507a55.jpg temp/1753861767_252643_1374415565_34ca151ee981bd874c67eac76e137323.jpg temp/1753861767_252643_1374415549_7579c6d1c0792c198ec38e366cafff5e.jpg temp/1753861767_252643_1374415546_a12828bfacec22525a53128d7f663932.jpg temp/1753861767_252643_1374415543_055934f0c4862f5cc12aaff797192ddb.jpg temp/1753861767_252643_1374415540_7fdc82c6bef41199d9d7854c8c300715.jpg temp/1753861767_252643_1374415536_9837b28a122a6a08f6d6549c02629c8c.jpg temp/1753861767_252643_1374415531_feb34c5ce6e56762962e69e771dc9e19.jpg temp/1753861767_252643_1374415517_99b78840503a47345d7d899dea4ad20d.jpg temp/1753861767_252643_1374415189_370b563f14300ed52374b4fc5d74a14e.jpg temp/1753861767_252643_1374415187_67dea572e065b1ac780dc3139e365581.jpg temp/1753861767_252643_1374415183_e0722fe2aa88cb0049351cfcdeb05afe.jpg temp/1753861767_252643_1374415178_cce6daad7aba935cb7a8a264d40f5df6.jpg temp/1753861767_252643_1374415115_9fe5f6768212421dc84e9bcfeac8d02d.jpg temp/1753861767_252643_1374415092_691838b3c705ed98f9c7cd46292c0a7e.jpg temp/1753861767_252643_1374415091_2aa228ee7d91082d31b8cbb0d0c30d90.jpg temp/1753861767_252643_1374415079_9cb78465fb18b2626dc5a4f4a5f91798.jpg temp/1753861767_252643_1374414559_f6ad40d37deab569e86fc651dd659dd9.jpg temp/1753861767_252643_1374414102_1e550b96e2ffc35d1b4349bc0801ad2b.jpg temp/1753861767_252643_1374414064_7e38f325dee09ad74dfe8e8b73523b2f.jpg temp/1753861767_252643_1374414031_cb05bdc34a469ee3969818b5842c5510.jpg temp/1753861767_252643_1374414001_269a4b2f10b693a8761f2ccd0558cf92.jpg temp/1753861767_252643_1374413937_7a7a682471b8cbdbdda352181e3c08fa.jpg temp/1753861767_252643_1374412633_5d6480d30143cfa51bdff99a194d7d83.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1374421544, 1374421510, 1374421474, 1374421441, 1374421407, 1374421390, 1374421359, 1374421357, 1374421355, 1374421352, 1374421350, 1374421259, 1374421251, 1374421217, 1374421213, 1374420883, 1374420882, 1374420880, 1374420877, 1374420875, 1374420871, 1374420830, 1374420826, 1374420823, 1374420782, 1374420781, 1374420754, 1374420750, 1374420747, 1374420744, 1374420561, 1374420556, 1374420344, 1374420342, 1374420220, 1374420216, 1374420212, 1374420211, 1374420208, 1374420207, 1374420205, 1374420177, 1374419999, 1374419996, 1374419992, 1374419990, 1374419988, 1374419983, 1374419959, 1374419955, 1374419935, 1374419898, 1374419864, 1374419827, 1374419235, 1374418681, 1374418312, 1374417824, 1374416810, 1374416724, 1374416661, 1374416576, 1374416515, 1374416429, 1374415718, 1374415715, 1374415713, 1374415708, 1374415681, 1374415679, 1374415677, 1374415674, 1374415671, 1374415593, 1374415589, 1374415585, 1374415567, 1374415565, 1374415549, 1374415546, 1374415543, 1374415540, 1374415536, 1374415531, 1374415517, 1374415189, 1374415187, 1374415183, 1374415178, 1374415115, 1374415092, 1374415091, 1374415079, 1374414559, 1374414102, 1374414064, 1374414031, 1374414001, 1374413937, 1374412633] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.05740690231323242 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 12.321377992630005 time spend to save output : 0.061728715896606445 total time spend for step 0 : 12.383106708526611 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 11.86user 3.13system 0:32.61elapsed 45%CPU (0avgtext+0avgdata 1972236maxresident)k 520inputs+244928outputs (6major+501408minor)pagefaults 0swaps