python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3071467 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 18228>100 please execute split_portfolio.py -i 22740043 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2977395'] with mtr_portfolio_ids : ['22740043'] and first list_photo_ids : [] new path : /proc/3071467/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 17.164891958236694 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu May 29 16:43:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-29 16:43:47.010135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-29 16:43:47.018419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 16:43:47.018628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 16:43:47.021356: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 16:43:47.023039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 16:43:47.023699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 16:43:47.026520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 16:43:47.027829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 16:43:47.034542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 16:43:47.035820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 16:43:47.036222: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-29 16:43:47.063117: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-29 16:43:47.064977: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f2c14000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-29 16:43:47.065001: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-29 16:43:47.286826: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x50e4f20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-29 16:43:47.286873: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-29 16:43:47.287769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 16:43:47.287863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 16:43:47.287892: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 16:43:47.287919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 16:43:47.287945: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 16:43:47.287971: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 16:43:47.287996: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 16:43:47.288022: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 16:43:47.289402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 16:43:47.289465: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 16:43:47.290335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-29 16:43:47.290360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-29 16:43:47.290374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-29 16:43:47.291904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3439 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.431859016418457 time used to load_weights : 0.5546965599060059 0it [00:00, ?it/s] 100it [00:00, 1620.50it/s]2025-05-29 16:43:58.312322: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 16:43:58.479991: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-29 16:43:58.509868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 16:43:58.514269: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-29 16:43:58.516322: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748529807_3071467_1361483127_a38e7c36a5fcea14951cc819275fbc9f_72001942.jpg temp/1748529807_3071467_1361483125_01bfd1d032c422096fd557ce4be3e4c0_81976477.jpg temp/1748529807_3071467_1361483124_deb592deb90f6e14b5c57b4a635d679c_24026658.jpg temp/1748529807_3071467_1361483123_387ea3423ffcc37b1b9db25305f1415e_1382206.jpg temp/1748529807_3071467_1361483122_9f4d96e2b7f65be9ff9127a0121b2c27_64215531.jpg temp/1748529807_3071467_1361483121_303d7bdf55411e47f823c00ab8283ba9_5435694.jpg temp/1748529807_3071467_1361483120_4130988fb113590626a6b75fe95023f6_19403298.jpg temp/1748529807_3071467_1361483119_993fa43af246447676a45f8b01c6ffbe_40506602.jpg temp/1748529807_3071467_1361483118_8c9d20be62d3a166a60a199ac4d974ac_72330811.jpg temp/1748529807_3071467_1361483117_31fee8884fed27ffe5495398af62af77_12770460.jpg temp/1748529807_3071467_1361483116_8c396ce2842ff957c6c2de9ec3450c95_71750507.jpg temp/1748529807_3071467_1361483115_8d97124e240918d4ad619a848f722cae_21924795.jpg temp/1748529807_3071467_1361483114_241174350480e205aac613cc857a1eeb_98491472.jpg temp/1748529807_3071467_1361483113_4b3b6c111daa16a1c4d53ef03f294dc7_55773772.jpg temp/1748529807_3071467_1361483112_81ba4357f2951d449e982865c718921a_42812992.jpg temp/1748529807_3071467_1361483111_19c61c13fadf5b519dd422636746ce52_53525364.jpg temp/1748529807_3071467_1361483110_06102d8d706e1ff6666a67413a88b8ae_96060551.jpg temp/1748529807_3071467_1361483104_d0359d9bd6756c4abeb2d6850173ec66_73681004.jpg temp/1748529807_3071467_1361483103_e436793f73f65c3e4d7b786b53fd80ea_58241627.jpg temp/1748529807_3071467_1361483102_31475aceb1eaaf28f1a0022a2d2ee180_61140137.jpg temp/1748529807_3071467_1361483101_6f630d626e23f4040e805543fd16c42c_14289306.jpg temp/1748529807_3071467_1361483100_9d32ee7ce374b761536d3f4b86f6ef90_40068220.jpg temp/1748529807_3071467_1361483099_c58248b982d5d8281cc55aae07a8395b_20996622.jpg temp/1748529807_3071467_1361483098_69b8ee5d44dbdc4c95a4ae357adcfdb5_55383307.jpg temp/1748529807_3071467_1361483097_e9ceee67789e69d8931ec280feb7d32a_76472605.jpg temp/1748529807_3071467_1361483096_beabef1e8c8681406481f1760f2159e5_37553228.jpg temp/1748529807_3071467_1361483095_71da4d88b6d7cbe8a43b1aa4231d23a9_52173872.jpg temp/1748529807_3071467_1361483094_5999dfb03ab0591cf995b5891025cb07_52774926.jpg temp/1748529807_3071467_1361483093_9e45d3c2a34ad09e740bb83563ed29ef_2853500.jpg temp/1748529807_3071467_1361483088_394f0740e537908c31e15336f1b7a994_26517403.jpg temp/1748529807_3071467_1361483087_8123c47e5396c9df078e40a1a66bbc05_74663980.jpg temp/1748529807_3071467_1361483086_ae2f8c38da2f0928c880696fb1a9037e_5980687.jpg temp/1748529807_3071467_1361483085_46196774a2a2cecdc5a0fcb788c476a4_65446521.jpg temp/1748529807_3071467_1361483084_2f67886998f57f79876abf1a7ca04dff_48392067.jpg temp/1748529807_3071467_1361483083_4a88678c5b038477a840720fb8064c8f_18286279.jpg temp/1748529807_3071467_1361483082_f04e819cdd59fc56d84b0fa86e6abf25_70279249.jpg temp/1748529807_3071467_1361483081_2189ecc82ed9af01896402746d4a9848_99387942.jpg temp/1748529807_3071467_1361483080_70a995954d468eaafc176167056be55e_74878823.jpg temp/1748529807_3071467_1361483079_277d167eb988920352a6a546082b4906_54307357.jpg temp/1748529807_3071467_1361483078_35e7afb8830298e781e115c668b1bf27_60019148.jpg temp/1748529807_3071467_1361483077_8443fda47857d0afcf92135d1aba2923_82008214.jpg temp/1748529807_3071467_1361483073_cdaef1522df2075990ab4d994acf549f_86342901.jpg temp/1748529807_3071467_1361483072_00b1513eb3c45d653f73ea27e8c55fb7_27407624.jpg temp/1748529807_3071467_1361483071_f4c52ada2aed4a182cc624a18de9c778_3703491.jpg temp/1748529807_3071467_1361483070_6deeb1720dbe1d4b467f4adc17ab0582_99071943.jpg temp/1748529807_3071467_1361483069_b30d6ece8563062154ba390e3865cac1_64760712.jpg temp/1748529807_3071467_1361483068_8dedb8055822d7ef2b9cc4e6135dd8ab_16447361.jpg temp/1748529807_3071467_1361483067_960eabec07dc3ec8c6621e9ebab608df_48855177.jpg temp/1748529807_3071467_1361483066_4311df619aa1807af3a25563e851345d_44163149.jpg temp/1748529807_3071467_1361483065_78753c2015967f352423dbfa165a8512_72403406.jpg temp/1748529807_3071467_1361483064_07849290f58d5a1ae120f2d05f4cb3da_29112924.jpg temp/1748529807_3071467_1361483063_714b0827dc08414130341a5263bd763f_12341527.jpg temp/1748529807_3071467_1361483062_0b3de51089a91f6d0f3e3fa7a91a0df0_25794141.jpg temp/1748529807_3071467_1361483058_9c9ba00cb455e4e6e8bbeaed0dff815c_19294239.jpg temp/1748529807_3071467_1361483057_0720f9b2defcb1c9197ddd78268259a3_88099730.jpg temp/1748529807_3071467_1361483056_8284a3ad296ebc17e5853a94f9b4e777_14778722.jpg temp/1748529807_3071467_1361483055_b1805cdc618e20ebcfa424bfbc8beea4_42233916.jpg temp/1748529807_3071467_1361483054_4ab1f1f1f81274b49c08c85ab2c0a2b5_30706427.jpg temp/1748529807_3071467_1361483053_42ef722b04ebd1a121110f00b1f0c57a_46270860.jpg temp/1748529807_3071467_1361483051_01ec596935f8868e0728dc2f6b7e9090_40426262.jpg temp/1748529807_3071467_1361483050_a76ad6a37ed8743626d8b07e3a1434a1_60659745.jpg temp/1748529807_3071467_1361483049_f06405f87167495868d21f535cbe8013_87462042.jpg temp/1748529807_3071467_1361483048_cc8816f34b8a39a131f9fc30962944d9_54098224.jpg temp/1748529807_3071467_1361483047_82f610b0d4a91d40dd4a68de99d5238e_70898682.jpg temp/1748529807_3071467_1361483046_4a0db74f97ddb132c9c2957e2bad7b06_11432920.jpg temp/1748529807_3071467_1361483045_2215c3db721971340f50e927b387d66d_87994878.jpg temp/1748529807_3071467_1361483044_b70b09861ff0da5c8ce8dfdb955e24b2_46721747.jpg temp/1748529807_3071467_1361483043_2ea5235b91c522e39ffa1d22105735d9_24749096.jpg temp/1748529807_3071467_1361483042_48671dc955d5415b1e590b983f99a914_11972088.jpg temp/1748529807_3071467_1361483041_6e598bd36f89a32fb77136cc8ff5bede_16812756.jpg temp/1748529807_3071467_1361483040_dcc064f08aa490a9fb6b35a70d87dfde_93016403.jpg temp/1748529807_3071467_1361483036_9d452c78500c5fba3b64b65765366191_68062388.jpg temp/1748529807_3071467_1361483035_cfbfbdc58e722ab85c0bcf6478a1ee6d_50130709.jpg temp/1748529807_3071467_1361483034_084ec650fdf538fe63752899ad904292_5351649.jpg temp/1748529807_3071467_1361483033_55ae55861752f5bafd5829df14ae817d_34394172.jpg temp/1748529807_3071467_1361483032_d02f9f4c3288725dfccaac3e45b4e9be_5098130.jpg temp/1748529807_3071467_1361483031_a2738d41c044716c87fa1c08b2197773_97915722.jpg temp/1748529807_3071467_1361483026_e1a84b4031967cdf305e658da09200b3_64536923.jpg temp/1748529807_3071467_1361483025_322cab8fb59e6bbc08904ea82e288d6e_91528489.jpg temp/1748529807_3071467_1361483024_74fcb492620ace5a2fd08d49222b9e3a_70103513.jpg temp/1748529807_3071467_1361483023_87af8b127499d1c53a12963e02c294f4_87095266.jpg temp/1748529807_3071467_1361483022_fc322cae66c3f2c8175425d917458061_24678367.jpg temp/1748529807_3071467_1361483021_88ac9cb466995e807ff59aef3be1fabe_15949063.jpg temp/1748529807_3071467_1361483019_6ade663ba729a1508743b156b3d67001_57948269.jpg temp/1748529807_3071467_1361483018_dc5ac095479b2c6dda95244ea8523c96_3694709.jpg temp/1748529807_3071467_1361483017_f3ac614876f1319b7233aa6aa483de83_62852243.jpg temp/1748529807_3071467_1361483016_b4decc0bf46b869c7a0e914e08686d41_1184290.jpg temp/1748529807_3071467_1361483015_cc3776812a775baf8f8bd1e9d8af1067_95805368.jpg temp/1748529807_3071467_1361483014_93f817aeba014c1474460665b9db82de_5385363.jpg temp/1748529807_3071467_1361483013_c693e1975dc5f893b12c9721bd7fb998_75172413.jpg temp/1748529807_3071467_1361483012_3483665f06bd57cd85408fae3388eaad_67786836.jpg temp/1748529807_3071467_1361483011_dbf260eb3ec77014a3af0c0c3d84ff4a_59519573.jpg temp/1748529807_3071467_1361483010_c2e7e05517a23629d5085788fba94a18_71012894.jpg temp/1748529807_3071467_1361483009_ab82b09b6dd8a380c3be7035106851d3_71889821.jpg temp/1748529807_3071467_1361483008_bdb0d7c17a1824fb4dc64824509d69a1_68330530.jpg temp/1748529807_3071467_1361483005_babfba5211c69ff701fb9772a4f213d2_9760665.jpg temp/1748529807_3071467_1361483004_2728e358f4d2f64a02b451c09ba56a20_78637755.jpg temp/1748529807_3071467_1361483002_3ed24b827e1aa8b7f2527b5457ee4dbe_59651240.jpg temp/1748529807_3071467_1361483001_da42a60c89e5203004b722c459f7e1c5_6209214.jpg temp/1748529807_3071467_1361483000_5d72d164d662008dd865b6a9ae919f60_89252715.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361483127, 1361483125, 1361483124, 1361483123, 1361483122, 1361483121, 1361483120, 1361483119, 1361483118, 1361483117, 1361483116, 1361483115, 1361483114, 1361483113, 1361483112, 1361483111, 1361483110, 1361483104, 1361483103, 1361483102, 1361483101, 1361483100, 1361483099, 1361483098, 1361483097, 1361483096, 1361483095, 1361483094, 1361483093, 1361483088, 1361483087, 1361483086, 1361483085, 1361483084, 1361483083, 1361483082, 1361483081, 1361483080, 1361483079, 1361483078, 1361483077, 1361483073, 1361483072, 1361483071, 1361483070, 1361483069, 1361483068, 1361483067, 1361483066, 1361483065, 1361483064, 1361483063, 1361483062, 1361483058, 1361483057, 1361483056, 1361483055, 1361483054, 1361483053, 1361483051, 1361483050, 1361483049, 1361483048, 1361483047, 1361483046, 1361483045, 1361483044, 1361483043, 1361483042, 1361483041, 1361483040, 1361483036, 1361483035, 1361483034, 1361483033, 1361483032, 1361483031, 1361483026, 1361483025, 1361483024, 1361483023, 1361483022, 1361483021, 1361483019, 1361483018, 1361483017, 1361483016, 1361483015, 1361483014, 1361483013, 1361483012, 1361483011, 1361483010, 1361483009, 1361483008, 1361483005, 1361483004, 1361483002, 1361483001, 1361483000] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.05976986885070801 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 14.384016990661621 time spend to save output : 0.07608628273010254 total time spend for step 0 : 14.460103273391724 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.28user 4.18system 0:35.02elapsed 49%CPU (0avgtext+0avgdata 1786692maxresident)k 49224inputs+159800outputs (339major+548461minor)pagefaults 0swaps