python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3619903 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7194>100 please execute split_portfolio.py -i 23429777 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2980365'] with mtr_portfolio_ids : ['23429777'] and first list_photo_ids : [] new path : /proc/3619903/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 12.860148429870605 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu May 29 19:10:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-29 19:10:45.992724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-29 19:10:46.023433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 19:10:46.023745: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 19:10:46.025919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 19:10:46.028112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 19:10:46.028442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 19:10:46.030715: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 19:10:46.031815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 19:10:46.036380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 19:10:46.037511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 19:10:46.038023: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-29 19:10:46.063095: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-29 19:10:46.065237: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa9e0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-29 19:10:46.065295: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-29 19:10:46.197272: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x43035cd0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-29 19:10:46.197336: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-29 19:10:46.198631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 19:10:46.198780: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 19:10:46.198821: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 19:10:46.198853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 19:10:46.198894: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 19:10:46.198943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 19:10:46.198976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 19:10:46.199007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 19:10:46.200473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 19:10:46.200553: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 19:10:46.201448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-29 19:10:46.201475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-29 19:10:46.201490: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-29 19:10:46.203078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 4028 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-29 19:10:53.530809: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.934569835662842 time used to load_weights : 0.1984391212463379 0it [00:00, ?it/s] 100it [00:00, 1597.30it/s]2025-05-29 19:10:57.476232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 19:10:57.684440: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-29 19:10:57.714940: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 19:10:57.735334: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-29 19:10:57.741150: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748538630_3619903_1361658934_f0e47f3193f2861acde8601708963d27.jpg temp/1748538630_3619903_1361658903_5e53e4bdf97899b172452b9b3916396e.jpg temp/1748538630_3619903_1361658875_e742e52a1691f3e08eb305d300842a38.jpg temp/1748538630_3619903_1361658844_5ed4ca915869e65a4614dc198f43d64e.jpg temp/1748538630_3619903_1361658812_97456c7d1ed21c2c3a322fa950b9a782.jpg temp/1748538630_3619903_1361658807_e0f40ba8b24ed1d262b581824b2f8b05.jpg temp/1748538630_3619903_1361658792_33ec67abb63669393431f73d409141d6.jpg temp/1748538630_3619903_1361658789_303afd1cc9ada31ad4635f7528bcb743.jpg temp/1748538630_3619903_1361658787_7b0e56146c6b3b9a491361c82bf3aa11.jpg temp/1748538630_3619903_1361658785_3c0ee7d3c2885de872e545d382757b9b.jpg temp/1748538630_3619903_1361658783_7399fa7ffdf27aaaf3e0bd4e8ac742a4.jpg temp/1748538630_3619903_1361658780_4adc88f156b6791b160f7f6f1468ab6f.jpg temp/1748538630_3619903_1361658545_9565213fb70cb53bafe7a01b9b5ae087.jpg temp/1748538630_3619903_1361658517_838f82e9ea48b0e3da00b71f4639bff7.jpg temp/1748538630_3619903_1361658489_76fc5853135947ef351966d86721f336.jpg temp/1748538630_3619903_1361658450_a590d8d01435d64c558392fc00a712ca.jpg temp/1748538630_3619903_1361658373_eed354db8a34d3fa296b560251065241.jpg temp/1748538630_3619903_1361658345_e83586a9066767d55365c9d757a55aaf.jpg temp/1748538630_3619903_1361658196_e935bc172ef0e1b3da5161d3d50d7474.jpg temp/1748538630_3619903_1361658169_112d67888384a6d52dc78d2363dc1b9f.jpg temp/1748538630_3619903_1361658141_33942c956d1f7cc2e4b117aee6309203.jpg temp/1748538630_3619903_1361658115_0b950f42c5460a6d12f8273102d1543f.jpg temp/1748538630_3619903_1361658088_290f0f72b29a1ae45c78c7af02e7fdcf.jpg temp/1748538630_3619903_1361658060_a2910b0acfd66ecd67d46194107e7b4e.jpg temp/1748538630_3619903_1361657948_9d6c97d55c43b8abfc0bbde609c1ca2d.jpg temp/1748538630_3619903_1361657937_18efa26f8a88d750ee23478908e62096.jpg temp/1748538630_3619903_1361657932_2f8c3f64671ba561b9c59cee32840fb3.jpg temp/1748538630_3619903_1361657928_1915c6bae1958d1cc64e0a51c8ba4e8c.jpg temp/1748538630_3619903_1361657923_64fe0f0a75c3d0b4b2e249232e7ca11e.jpg temp/1748538630_3619903_1361657920_59a8868d4087add55d60ad0a3abcd2e5.jpg temp/1748538630_3619903_1361657914_f23e8983ca794ff9244cc19fc2de11dc.jpg temp/1748538630_3619903_1361657912_4888b2a51b598fb352f9da05b678915e.jpg temp/1748538630_3619903_1361657910_8e94d0f49e1b196b5b9e807b79ff3bee.jpg temp/1748538630_3619903_1361657907_1b11480edb0075097f4d355a53dca843.jpg temp/1748538630_3619903_1361657905_699d70ab70b6c99718887f1680beaa46.jpg temp/1748538630_3619903_1361657893_bed65e31b3c5a80486479784f66ff971.jpg temp/1748538630_3619903_1361657699_2ce801b0c46c8eaad3f51f7d60d0d71e.jpg temp/1748538630_3619903_1361657625_6e8597c54b19681db8d9e481045b4082.jpg temp/1748538630_3619903_1361657600_30e9f8a5c2600ce3a547d66880835e15.jpg temp/1748538630_3619903_1361657575_400f60cf36dd4b649abd89ffa0866a72.jpg temp/1748538630_3619903_1361657573_5670a6720396bc3c0b393f9ee19fed3d.jpg temp/1748538630_3619903_1361657558_2e932d7475d7996d7bc30dc70f2f3967.jpg temp/1748538630_3619903_1361657481_7b560a3b3f82653f259ca328f1622564.jpg temp/1748538630_3619903_1361657450_8bd1cd1b2b28d5f12ab5cafedcd29b17.jpg temp/1748538630_3619903_1361657424_9004f5bda8a743c941f4cc2fca2540d6.jpg temp/1748538630_3619903_1361657404_546e0e44b07841a8a26db639b6bfc27a.jpg temp/1748538630_3619903_1361657401_ac4c056e6d30f948261ca1910dfb2230.jpg temp/1748538630_3619903_1361657397_78db3de95054283309ff33b4aa42fa3a.jpg temp/1748538630_3619903_1361656763_ed4ee41324af84ec08a9a86b8ee28efe.jpg temp/1748538630_3619903_1361656731_b8a2f76f4aa706e20d19fa8c09987694.jpg temp/1748538630_3619903_1361656703_8ce9c5837a782d55dab0d72f565f0366.jpg temp/1748538630_3619903_1361656678_cd8269fe5cba5b85e902e55c79079470.jpg temp/1748538630_3619903_1361656668_d3e26d39715aea222a4fb4a287cd7aa9.jpg temp/1748538630_3619903_1361656643_ddf4a0d8bbedd80018a6bb3d359f5805.jpg temp/1748538630_3619903_1361656599_f1a7bd887b6dc7b4be493fa79bc10b0b.jpg temp/1748538630_3619903_1361656593_a941a9d27c2cdb8afed81eca70dd18b9.jpg temp/1748538630_3619903_1361656569_fa3733f38fc0fe3de83fa951ab6e9761.jpg temp/1748538630_3619903_1361656540_58bb9cabdfa18b8b64bbcd098ef5268d.jpg temp/1748538630_3619903_1361656510_9da04cf4b52c0bbecb8293df223a8661.jpg temp/1748538630_3619903_1361656498_29d33178a2defa1734eee6824d863b9c.jpg temp/1748538630_3619903_1361656484_1ef0040a3d3fe1866e5a4ecb18dafdc9.jpg temp/1748538630_3619903_1361656481_b73e5fefe42c41227b4e694ccfbdfa01.jpg temp/1748538630_3619903_1361656476_c2874930adb4066587e91c941614ed1f.jpg temp/1748538630_3619903_1361656471_20c05266fa048215099ddc422f0acb4f.jpg temp/1748538630_3619903_1361656459_a1d5e58c42ce8bbb83d50e4a35e19122.jpg temp/1748538630_3619903_1361656431_1599740e30f9b2a94940ae172c703479.jpg temp/1748538630_3619903_1361656302_6ae5db265b884beba39880bbe50fe9ab.jpg temp/1748538630_3619903_1361656272_57a839f13c3a2ccbbfd685626ac151a4.jpg temp/1748538630_3619903_1361656244_44ce2c5205f36ff42f0346e81ca9b7d1.jpg temp/1748538630_3619903_1361656214_7921b86a3650ec33f5bdb19b328dffec.jpg temp/1748538630_3619903_1361656181_a59d8e4a7e0f2665870716a27fa3919f.jpg temp/1748538630_3619903_1361656152_e5802722a972dfc3e3576a878b2e20bf.jpg temp/1748538630_3619903_1361655691_400334b7841da5f851559c4fbd024d9b.jpg temp/1748538630_3619903_1361655688_5f165167f3d6014d5bf6f94fc0aa9f26.jpg temp/1748538630_3619903_1361655686_e23609f9ef9cc77f1015c5e8a8b6f1af.jpg temp/1748538630_3619903_1361655685_2918d86deb2d044d21298bd686c8b2fc.jpg temp/1748538630_3619903_1361655682_49da6ff0f9dc5fc9a6cef2969c1d3215.jpg temp/1748538630_3619903_1361655680_18bb18a4a70cc0917b4fb3385e221d25.jpg temp/1748538630_3619903_1361655672_f2c92fed3660808069f1e069b664a460.jpg temp/1748538630_3619903_1361655670_09eb92c77f0e565a3d47795bd6f80a78.jpg temp/1748538630_3619903_1361655667_b5b62b05ce857de6e0fddc26abee7035.jpg temp/1748538630_3619903_1361655662_75c2b280b60947812992483279970944.jpg temp/1748538630_3619903_1361655657_397e8b2e6fe87c65b0bd8774afaee910.jpg temp/1748538630_3619903_1361655654_278740389148e8bad9c337147c7fc74e.jpg temp/1748538630_3619903_1361655648_3a074304356960627235c0fd986d3a43.jpg temp/1748538630_3619903_1361655646_b4291c87570395cb1901608499ae3eba.jpg temp/1748538630_3619903_1361655644_d10438292c388510587c9d4ce8868a06.jpg temp/1748538630_3619903_1361655641_46258f0b32e319397662e6e0ebdea8fa.jpg temp/1748538630_3619903_1361655637_92a609993b9dc55e6ddf1c6d57c66a7b.jpg temp/1748538630_3619903_1361655632_00bebcd590cb3c219365e84a7761e59a.jpg temp/1748538630_3619903_1361655615_f1ef3dd94504121b4dbd67e5405ed3cd.jpg temp/1748538630_3619903_1361655614_be1ef9390c9fba01d24b260260d12c80.jpg temp/1748538630_3619903_1361655613_819c2f7f413b8184fd713a353300dc1f.jpg temp/1748538630_3619903_1361655611_0b0701ff2c6afd67b0ab8cae81dda1ef.jpg temp/1748538630_3619903_1361655609_d7668a948e3beb848d44bc7fdad7a133.jpg temp/1748538630_3619903_1361655606_1d227f12f94f0d277a4d057ca864327f.jpg temp/1748538630_3619903_1361655598_95e8261fd97c51e947d18936ef6e017a.jpg temp/1748538630_3619903_1361655593_f4e8cdb47e0f29ff129cf3e129119b81.jpg temp/1748538630_3619903_1361655589_aef7f0ba143ba57a2843ebdf25476f67.jpg temp/1748538630_3619903_1361655585_4b9d50e8c49663f596770f2224382c68.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361658934, 1361658903, 1361658875, 1361658844, 1361658812, 1361658807, 1361658792, 1361658789, 1361658787, 1361658785, 1361658783, 1361658780, 1361658545, 1361658517, 1361658489, 1361658450, 1361658373, 1361658345, 1361658196, 1361658169, 1361658141, 1361658115, 1361658088, 1361658060, 1361657948, 1361657937, 1361657932, 1361657928, 1361657923, 1361657920, 1361657914, 1361657912, 1361657910, 1361657907, 1361657905, 1361657893, 1361657699, 1361657625, 1361657600, 1361657575, 1361657573, 1361657558, 1361657481, 1361657450, 1361657424, 1361657404, 1361657401, 1361657397, 1361656763, 1361656731, 1361656703, 1361656678, 1361656668, 1361656643, 1361656599, 1361656593, 1361656569, 1361656540, 1361656510, 1361656498, 1361656484, 1361656481, 1361656476, 1361656471, 1361656459, 1361656431, 1361656302, 1361656272, 1361656244, 1361656214, 1361656181, 1361656152, 1361655691, 1361655688, 1361655686, 1361655685, 1361655682, 1361655680, 1361655672, 1361655670, 1361655667, 1361655662, 1361655657, 1361655654, 1361655648, 1361655646, 1361655644, 1361655641, 1361655637, 1361655632, 1361655615, 1361655614, 1361655613, 1361655611, 1361655609, 1361655606, 1361655598, 1361655593, 1361655589, 1361655585] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.049957275390625 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 14.707340717315674 time spend to save output : 0.05810666084289551 total time spend for step 0 : 14.76544737815857 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.53user 3.74system 0:34.30elapsed 53%CPU (0avgtext+0avgdata 1844580maxresident)k 1696inputs+168320outputs (0major+558305minor)pagefaults 0swaps