python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3546441 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 14262>100 please execute split_portfolio.py -i 22866006 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2818761'] with mtr_portfolio_ids : ['22866006'] and first list_photo_ids : [] new path : /proc/3546441/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 13.44604229927063 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Tue Sep 2 17:10:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-09-02 17:10:44.714559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-02 17:10:44.744372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-02 17:10:44.744713: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-02 17:10:44.747021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-02 17:10:44.749032: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-02 17:10:44.749408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-02 17:10:44.751714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-02 17:10:44.752833: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-02 17:10:44.757800: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-02 17:10:44.759180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-02 17:10:44.759637: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-02 17:10:44.788652: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-02 17:10:44.790879: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbf70000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-02 17:10:44.790949: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-02 17:10:44.908905: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xf0e34d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-02 17:10:44.908967: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-02 17:10:44.910188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-02 17:10:44.910291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-02 17:10:44.910323: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-02 17:10:44.910349: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-02 17:10:44.910375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-02 17:10:44.910399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-02 17:10:44.910425: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-02 17:10:44.910451: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-02 17:10:44.912479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-02 17:10:44.912625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-02 17:10:44.913677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-02 17:10:44.913702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-02 17:10:44.913717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-02 17:10:44.915512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 6019 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.052296161651611 time used to load_weights : 0.27100062370300293 0it [00:00, ?it/s] 30it [00:00, 288.77it/s] 79it [00:00, 403.92it/s] 100it [00:00, 381.81it/s]2025-09-02 17:10:56.685937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-02 17:10:56.946220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-02 17:10:57.032390: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-09-02 17:10:57.034860: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1756825828_3546441_1380879614_a804f4b3f1bfc89f40c9d4de26a8733b_39201940.jpg temp/1756825828_3546441_1380879613_a28b3384253981570b4cd2a172091ea4_56254503.jpg temp/1756825828_3546441_1380879612_aa06d78aa49311452a8ddea70829e0ac_83683450.jpg temp/1756825828_3546441_1380879611_fcb768e5fb44a2006f33838aa3aae240_35830464.jpg temp/1756825828_3546441_1380879610_d035368a445e205096fa05848b7a5031_29884061.jpg temp/1756825828_3546441_1380879609_e86355875c122c30af35fe2526fc5b12_24419231.jpg temp/1756825828_3546441_1380879509_83511f68bb8974f746e01150a61038ba_54537934.jpg temp/1756825828_3546441_1380879508_e69e4a4025eadf4f8990acac02351042_18461566.jpg temp/1756825828_3546441_1380879507_27ce9518b9548148e2fab153007850e7_98737467.jpg temp/1756825828_3546441_1380879505_130c7a17a1e903814284b4010ce79be6_76601319.jpg temp/1756825828_3546441_1380879504_b5beab0f2e6ae6c93d013c127be7251a_62496956.jpg temp/1756825828_3546441_1380879503_bab55c5bd541ce2256c388c390e46469_85825802.jpg temp/1756825828_3546441_1380879271_d26fd1b0bde02c356334ede30aeeea94_33557391.jpg temp/1756825828_3546441_1380879268_83c405be8c5d32e549c9b2cde6ab6657_36445145.jpg temp/1756825828_3546441_1380879265_835189d3caa8eb9c02628f5d6a7cc7ec_53992790.jpg temp/1756825828_3546441_1380879261_0f803bed60136d735a223800aa10d985_83079841.jpg temp/1756825828_3546441_1380879257_418798a74580b7b9f65ba5e02e6138a2_85377261.jpg temp/1756825828_3546441_1380879254_e70a0cc8c1a24b31f50fd9f19a4c76d7_63367897.jpg temp/1756825828_3546441_1380879008_28cc1d64c802590ef6fe64a99df0813d_13179845.jpg temp/1756825828_3546441_1380879005_6cb570923782b6de6bd1e6de674087d3_20982932.jpg temp/1756825828_3546441_1380879001_6fa407df8a93e27b728c6d5444ef176b_23995041.jpg temp/1756825828_3546441_1380878998_394829707c5457aaa0dd8cb290502ec9_29830250.jpg temp/1756825828_3546441_1380878995_79ca71c7cb4234e33daa6d0187605816_65896905.jpg temp/1756825828_3546441_1380878992_45ef4aa0e73ffc3b169356c4ceb774a5_94216269.jpg temp/1756825828_3546441_1380878806_5f574cacb60d562dc6fa3abbe381d349_86067473.jpg temp/1756825828_3546441_1380878804_39b46d1ed4a80384dffd97d2368a6529_77189345.jpg temp/1756825828_3546441_1380878802_24e1d0798ec416af939185c2c718a661_32051703.jpg temp/1756825828_3546441_1380878800_cec9eb7110406e4614d549d3444c64ee_38489073.jpg temp/1756825828_3546441_1380878798_5d88c0a1f3b1de45bb946d4904e6aba0_31026994.jpg temp/1756825828_3546441_1380878796_552e4de7f025e332325205d7a40518df_94165568.jpg temp/1756825828_3546441_1380878728_412910cf1223f02ce1ea168e1ff58e7a_77147630.jpg temp/1756825828_3546441_1380878726_642701e6e077909ef895a58732bb43d7_65341893.jpg temp/1756825828_3546441_1380878724_22414fb0d251fa31cba6f90532597187_77378962.jpg temp/1756825828_3546441_1380878722_f6e4faa46a99d5affa3468f67ef9740d_39621646.jpg temp/1756825828_3546441_1380878720_29e44161e81fd14db91bab2120e54a25_35132347.jpg temp/1756825828_3546441_1380878718_c30e2dda5ac10bf0cc89df04186acd26_99351658.jpg temp/1756825828_3546441_1380878652_9b4caa43c0ac80a1195797ff32404aa7_52959221.jpg temp/1756825828_3546441_1380878651_c9c97ec87d621921e328a64a3b9026dc_32599026.jpg temp/1756825828_3546441_1380878650_8c20c1240566ac486105db34f738ceab_27904889.jpg temp/1756825828_3546441_1380878649_2c1ce33910ef7cc2e680f0f17b2a72d1_87168874.jpg temp/1756825828_3546441_1380878648_fcbb2bb46e6050bdba547d4ae2fd2b58_86344430.jpg temp/1756825828_3546441_1380878647_a550fe1f1dab5b0b46caadfe15411cd1_82559173.jpg temp/1756825828_3546441_1380878544_0bef61ef8b96e0ecd36c5f30bfda6cca_77636925.jpg temp/1756825828_3546441_1380878543_dda89476eb54bdc866d21f5a25c25800_21398991.jpg temp/1756825828_3546441_1380878542_01040929d970f80c6b807dfb069c4859_24197740.jpg temp/1756825828_3546441_1380878541_b26e39254b0ae19268988467f03939fb_29119368.jpg temp/1756825828_3546441_1380878540_2433edf40c6b3620811e43b5e363682f_11955685.jpg temp/1756825828_3546441_1380878538_fdb394b7b5361f9c8d456f4670f85dae_57567616.jpg temp/1756825828_3546441_1380878524_acd26dd72983e8a4bb3db51168ac5108_44763562.jpg temp/1756825828_3546441_1380878523_246870f9e7582da76b446a2559841bc5_47302932.jpg temp/1756825828_3546441_1380878522_a6072aeb380c5606eebec2e5c0da370b_21570198.jpg temp/1756825828_3546441_1380878521_800cfce7073136fa51c9d00be45b592c_2761611.jpg temp/1756825828_3546441_1380878520_0cc7b1a150ea9fa2d7fc5c900c4b511d_51717163.jpg temp/1756825828_3546441_1380878519_c0e81fa6e7b4ddcf53e3396609b9b216_67234455.jpg temp/1756825828_3546441_1380878509_d07028141f3199c8c5a1f256416c14fa_80114767.jpg temp/1756825828_3546441_1380878508_6f28814c275625c6d2824b10bdcc4588_56662043.jpg temp/1756825828_3546441_1380878507_96eeb9842ac39a92027f818765611a69_4551186.jpg temp/1756825828_3546441_1380878506_1ea276a1f2d53a86b84cea88f8cf042a_23808992.jpg temp/1756825828_3546441_1380878505_3b1866cbf7ca3fafbff0c38f497ff448_65046873.jpg temp/1756825828_3546441_1380878504_08fe90befd281991dbc5c9cf08691025_38082038.jpg temp/1756825828_3546441_1380878487_2fee7c77463ac31aa7f97145e4e6e327_6247883.jpg temp/1756825828_3546441_1380878486_e918555aff3831b04ff29d58955f95ba_63124818.jpg temp/1756825828_3546441_1380878485_bd8c558b0ad56281cb731f1cac450f2b_94772721.jpg temp/1756825828_3546441_1380878484_5da8d8670016494958c937ed5557eb0e_86820721.jpg temp/1756825828_3546441_1380878483_d6daa0b40786ae98b8368ec72236172e_84576171.jpg temp/1756825828_3546441_1380878482_d42746b787626888473c20d1f74dcc1f_65872334.jpg temp/1756825828_3546441_1380878412_b2dfa2d5861d70335ee5ef0cc31d8762_57474707.jpg temp/1756825828_3546441_1380878410_4a0ab7a088cc1732f828e3c8c487fe9a_56864173.jpg temp/1756825828_3546441_1380878408_8c24a0781d427b84bb68d6c3465a8f51_58434358.jpg temp/1756825828_3546441_1380878405_48bb08d3e5a6656dd3ce81f66b8ae7b5_45488020.jpg temp/1756825828_3546441_1380878402_dbc05bfce4ff05d4a1056b25dc839d82_69347630.jpg temp/1756825828_3546441_1380878399_4fd0cf19390fc9dcfb0f9b0b50ba7ac2_62074321.jpg temp/1756825828_3546441_1380878140_6ce269f30f037593fb6943986f758016_33314751.jpg temp/1756825828_3546441_1380878138_e1d68c9f17edb79ec8ef69204dd0a931_91043604.jpg temp/1756825828_3546441_1380878136_d6f3c17c272df586be42660ccf46c434_23970416.jpg temp/1756825828_3546441_1380878134_bb0e56e268d797ff93dfc1406b9b36a1_35032066.jpg temp/1756825828_3546441_1380878132_d4458ad240819fe98df77dc5a84e8028_69391450.jpg temp/1756825828_3546441_1380878130_67981bdf09dd54d255713a1b811bf16f_58529220.jpg temp/1756825828_3546441_1380878054_a2858ef9a78c407864ac5d0b47e33c47_59119412.jpg temp/1756825828_3546441_1380878053_3d8f4516cd2ffe1bdad897de7725bb94_94668388.jpg temp/1756825828_3546441_1380878052_36b51e20a9dfd7417780f099d246a26d_98740529.jpg temp/1756825828_3546441_1380878051_81618c6ddff837d2eddd10f950c5039a_76296497.jpg temp/1756825828_3546441_1380878050_ccdbc34ed3243a4760c7907d0f0f43fc_42125732.jpg temp/1756825828_3546441_1380878049_d09e05205d71368dab3aa7f52e1ddd13_32132124.jpg temp/1756825828_3546441_1380877982_3b53163ac9d3a42957a65470b918472e_89664928.jpg temp/1756825828_3546441_1380877980_c361663af11b701a4d041e31d6205dd6_46867194.jpg temp/1756825828_3546441_1380877978_8392c56cf0e2bd0753721ec4e23c7ff1_28142970.jpg temp/1756825828_3546441_1380877976_f5d4c19eea004220026f175b3ba1d033_76957599.jpg temp/1756825828_3546441_1380877973_0ff1d5370c0d17b389a2b4c56673df4f_93381614.jpg temp/1756825828_3546441_1380877970_231f96185951b03a4fb619b55d363007_58609764.jpg temp/1756825828_3546441_1380877805_d2171369264ebdf8dbbcc989f1c6e2c3_59870462.jpg temp/1756825828_3546441_1380877802_affbba605fc5c173d5dd122777c32e49_66045050.jpg temp/1756825828_3546441_1380877799_b9289f6cf00ae59733a63572cc33bd0f_15079676.jpg temp/1756825828_3546441_1380877796_3bf95b9026c7a1d4e466ee8d034ae08e_70580342.jpg temp/1756825828_3546441_1380877793_4e7a6bf18d153b7d1a914fcd8f3ddce6_82886953.jpg temp/1756825828_3546441_1380877790_b70255f9f269572749d8834f16fae13f_29970041.jpg temp/1756825828_3546441_1380877608_046ae9666b363a3dae564a7de5f3bc44_82052599.jpg temp/1756825828_3546441_1380877606_412f71afe95242809ed1e375baa9447c_5859487.jpg temp/1756825828_3546441_1380877604_23f1e5332db97c86c9937d455fb341b2_85412457.jpg temp/1756825828_3546441_1380877602_aff8a2c1511b5976eca7c1cdc8b12cff_890883.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1380879614, 1380879613, 1380879612, 1380879611, 1380879610, 1380879609, 1380879509, 1380879508, 1380879507, 1380879505, 1380879504, 1380879503, 1380879271, 1380879268, 1380879265, 1380879261, 1380879257, 1380879254, 1380879008, 1380879005, 1380879001, 1380878998, 1380878995, 1380878992, 1380878806, 1380878804, 1380878802, 1380878800, 1380878798, 1380878796, 1380878728, 1380878726, 1380878724, 1380878722, 1380878720, 1380878718, 1380878652, 1380878651, 1380878650, 1380878649, 1380878648, 1380878647, 1380878544, 1380878543, 1380878542, 1380878541, 1380878540, 1380878538, 1380878524, 1380878523, 1380878522, 1380878521, 1380878520, 1380878519, 1380878509, 1380878508, 1380878507, 1380878506, 1380878505, 1380878504, 1380878487, 1380878486, 1380878485, 1380878484, 1380878483, 1380878482, 1380878412, 1380878410, 1380878408, 1380878405, 1380878402, 1380878399, 1380878140, 1380878138, 1380878136, 1380878134, 1380878132, 1380878130, 1380878054, 1380878053, 1380878052, 1380878051, 1380878050, 1380878049, 1380877982, 1380877980, 1380877978, 1380877976, 1380877973, 1380877970, 1380877805, 1380877802, 1380877799, 1380877796, 1380877793, 1380877790, 1380877608, 1380877606, 1380877604, 1380877602] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.047258853912353516 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 15.425397157669067 time spend to save output : 0.05597257614135742 total time spend for step 0 : 15.481369733810425 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.79user 4.68system 0:34.21elapsed 56%CPU (0avgtext+0avgdata 1914296maxresident)k 376720inputs+192856outputs (239major+609625minor)pagefaults 0swaps