python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1298767 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 803>100 please execute split_portfolio.py -i 21130749 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2646420'] with mtr_portfolio_ids : ['21130749'] and first list_photo_ids : [] new path : /proc/1298767/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 94 ; length of list_pids : 94 ; length of list_args : 94 time to download the photos : 13.022668838500977 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri Mar 7 06:46:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 35 wait 20 seconds inside check gpu memory 2025-03-07 06:47:05.327553: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-07 06:47:05.360636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 06:47:05.361760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 06:47:05.366093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 06:47:05.369775: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 06:47:05.371338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 06:47:05.375563: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 06:47:05.377220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 06:47:05.384778: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 06:47:05.386246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 06:47:05.386667: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-07 06:47:05.419291: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-07 06:47:05.421623: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f435c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-07 06:47:05.421672: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-07 06:47:05.669451: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4324e520 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-07 06:47:05.669525: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-07 06:47:05.670729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 06:47:05.670846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 06:47:05.670881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 06:47:05.670932: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 06:47:05.670967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 06:47:05.671000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 06:47:05.671035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 06:47:05.671072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 06:47:05.672567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 06:47:05.672648: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 06:47:05.673559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 06:47:05.673584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 06:47:05.673598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 06:47:05.675265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3630 max_wait_temp : 2 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.8491456508636475 time used to load_weights : 0.48948073387145996 0it [00:00, ?it/s] 21it [00:00, 202.36it/s] 42it [00:00, 197.50it/s] 62it [00:00, 195.06it/s] 82it [00:00, 194.06it/s] 94it [00:00, 182.34it/s]2025-03-07 06:47:16.704981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 06:47:16.903219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 06:47:17.440599: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-03-07 06:47:17.442592: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1741326387_1298767_1341402794_3cf4515cde160320e8e903eafed93c99.jpg temp/1741326387_1298767_1341402791_51ce119bb21991cbcdb2a69567bf0559.jpg temp/1741326387_1298767_1341402761_f42fd9d91effd9e7190a390d94243caf.jpg temp/1741326387_1298767_1341402756_225432f7adf1813ff9b54931ef90ee39.jpg temp/1741326387_1298767_1341402570_151f72abd24a11646168e89c330dbf06.jpg temp/1741326387_1298767_1341402566_b576ea29b86ae4e3fbf435e6c2823d43.jpg temp/1741326387_1298767_1341402542_21598af138f974e3f0da7e6af35e033b.jpg temp/1741326387_1298767_1341402538_430812c81e9fb1ef88deb7aa80b548ea.jpg temp/1741326387_1298767_1341402473_554849b90d2cfffcf3b9d5ca560b1f5c.jpg temp/1741326387_1298767_1341402468_f6570fe373b3827e6c5c7293bdc522df.jpg temp/1741326387_1298767_1341402412_15531a35667eae5bf4640751c345bf35.jpg temp/1741326387_1298767_1341402408_47f901145f5c153f2d2e12d6f0612596.jpg temp/1741326387_1298767_1341402300_8bf36db1c82e8eb92f9852ca288fcadd.jpg temp/1741326387_1298767_1341402288_86b167873cfcdf98f6a80e974c4a6d48.jpg temp/1741326387_1298767_1341402196_bb80c4b7d1db5d0ccc839ee27a716d9c.jpg temp/1741326387_1298767_1341402193_6627f57a3d36f88de4d9b34f217b9fef.jpg temp/1741326387_1298767_1341402143_314880f52533903b898b4c21510e6939.jpg temp/1741326387_1298767_1341402140_d5f9319c14a3a5ef8111a9534834a3a1.jpg temp/1741326387_1298767_1341402108_9a083ddfd8432031e5748158f6f11ddd.jpg temp/1741326387_1298767_1341402101_9b5723f5432174b97e87a0df60bcf6d1.jpg temp/1741326387_1298767_1341401885_dad1d12b53001168126858a6e31f8a68.jpg temp/1741326387_1298767_1341401882_5e41aecba9342144b890626358daaa6b.jpg temp/1741326387_1298767_1341401771_c6173f0b31b40515129cdd3a8f541ca7.jpg temp/1741326387_1298767_1341401768_860616975f2933c7e7dea3430f26a9f7.jpg temp/1741326387_1298767_1341401664_179a58fb8ce6f6d8e103e378e1f04a13.jpg temp/1741326387_1298767_1341401662_bdb089a1bb5b3f8a16d9572f873076f8.jpg temp/1741326387_1298767_1341401304_26a69143f4c52e828474640275ab0872.jpg temp/1741326387_1298767_1341401302_1e0543c6b2eb08719de362d319685c94.jpg temp/1741326387_1298767_1341401162_01a74e999d5aa9ea9e0a91b7848755c6.jpg temp/1741326387_1298767_1341401160_e7903c7c709ef64f5f22c32cc1ec008e.jpg temp/1741326387_1298767_1341401072_84488bc52feba552e658a659bfc10ee2.jpg temp/1741326387_1298767_1341401067_b59a996fffddae57d2106aeaaedef412.jpg temp/1741326387_1298767_1341400985_46f5f33797c0581b867173001251be18.jpg temp/1741326387_1298767_1341400966_d7e54d0df54691cf941f25a9fcb0e015.jpg temp/1741326387_1298767_1341400604_f779362cf79aac8948c513d5c3ae6411.jpg temp/1741326387_1298767_1341400600_19b77987fc98b8765d29a636cdd68517.jpg temp/1741326387_1298767_1341400522_a230dfb630ee125beb08c14b38a07812.jpg temp/1741326387_1298767_1341400518_d3dd58cc722b1b7ec831e5d13087e3e2.jpg temp/1741326387_1298767_1341400385_880de1f9ec36cc6cc3180723421318bf.jpg temp/1741326387_1298767_1341400382_5e42a7ba20704ab3fbd10a71b1423975.jpg temp/1741326387_1298767_1341400337_5659f4c868a232bd51568e42da3aba32.jpg temp/1741326387_1298767_1341400332_71356349e8f7788a4ec4d8824ee3609c.jpg temp/1741326387_1298767_1341399934_391266337ebc7879d30937dea6d38869.jpg temp/1741326387_1298767_1341399930_252e90e50a86881b91862d6bcc1f7ac8.jpg temp/1741326387_1298767_1341399864_227fc2f7002214f6f6b2281cdf65d5c2.jpg temp/1741326387_1298767_1341399858_28ce5d8e41dc0c4570f5629b7bb7a214.jpg temp/1741326387_1298767_1341398981_602e225390d70d01dc43f24f2ea01c57.jpg temp/1741326387_1298767_1341398977_5bbf02bb72c07fdf29ed768efbb1f8c9.jpg temp/1741326387_1298767_1341398911_f82541685b202cf2636709a77fbb543e.jpg temp/1741326387_1298767_1341398897_7f938cc806e9051887e178bccaecc2b4.jpg temp/1741326387_1298767_1341398742_cff5e4811fe5c0df13d9b9ed75696554.jpg temp/1741326387_1298767_1341398732_5564421d348b08ccbfb3e2b88d9697e7.jpg temp/1741326387_1298767_1341398683_55b061e1fdfd5052730cc5f693b241d0.jpg temp/1741326387_1298767_1341398679_160788b4b791558e6e5457881d942a56.jpg temp/1741326387_1298767_1341398622_dc5bd4c457b32684745250131e039712.jpg temp/1741326387_1298767_1341398616_72c780726e2467a5ee2adaee7553b2a4.jpg temp/1741326387_1298767_1341398515_cab64a9a6b004520a93d5b0f797e9550.jpg temp/1741326387_1298767_1341398509_d62e4769d9926578144da16aac3eb13e.jpg temp/1741326387_1298767_1341398365_cc80171f53aba1860967e030c0634d93.jpg temp/1741326387_1298767_1341398347_b88c3cb563c315bcc784663b37a35f58.jpg temp/1741326387_1298767_1341398197_6fcd3b90f8856d619ac9772ef2884f0c.jpg temp/1741326387_1298767_1341398193_d7769fb497624f7fccb9e7f2bbde3f28.jpg temp/1741326387_1298767_1341398146_3e447fccbcbbe7333491856b0de12e10.jpg temp/1741326387_1298767_1341398140_5a9188858cfd9ac91ecc5a78ac1f6a37.jpg temp/1741326387_1298767_1341398077_a4e9957bf10974d7b254f98b61dee932.jpg temp/1741326387_1298767_1341398073_d731a029c0d214d5d9ebcd2621b0fe03.jpg temp/1741326387_1298767_1341398052_d1c8e0c17d768465cd73b803655ab781.jpg temp/1741326387_1298767_1341398050_b63680384b97f2a12a9039debf8fd5d4.jpg temp/1741326387_1298767_1341398025_820271ef09b63011163cbd19d1b8507a.jpg temp/1741326387_1298767_1341398021_67e2a52956b60c9128eaa5b04980adca.jpg temp/1741326387_1298767_1341398013_b77dc8e4ef13efe341dbf8a5fda222f0.jpg temp/1741326387_1298767_1341398011_e0307de7e6f1585f3c4fc998294e7d12.jpg temp/1741326387_1298767_1341397869_bf6625c10de236c58bf5f7b44d870f2c.jpg temp/1741326387_1298767_1341397865_8f1062372be628867839e166a9644b23.jpg temp/1741326387_1298767_1341397840_f2e9c6ea51efc6a9235aa99098b78ab9.jpg temp/1741326387_1298767_1341397836_8443635c88f87b80a51c5124beb9612b.jpg temp/1741326387_1298767_1341397795_7cdc19c138f6a4b26a2e6da990590980.jpg temp/1741326387_1298767_1341397792_fabe74192ec32f754f2a306cc5bc7945.jpg temp/1741326387_1298767_1341397769_c1f4b56e21f4bd9e0a66b132bce8abaa.jpg temp/1741326387_1298767_1341397765_400fe69bccd90c3f7573adeb76be987b.jpg temp/1741326387_1298767_1341397709_89abc9018b498ece06e3b156a666be93.jpg temp/1741326387_1298767_1341397706_ad09337359b900c043894742f6bdae23.jpg temp/1741326387_1298767_1341397523_6ce5cb4e58f28df59994de2673fddd2b.jpg temp/1741326387_1298767_1341397517_0bdcdb5a9e9f3d542a2abbb9a2fbe63d.jpg temp/1741326387_1298767_1341397322_0bf9f5908e92fb1e06259d634b8c3e07.jpg temp/1741326387_1298767_1341397259_c40d178ed6d9d0de0cd4bbb41ce9ad92.jpg temp/1741326387_1298767_1341396868_3f7467331408f74024431db4e223e302.jpg temp/1741326387_1298767_1341396863_10ba3caf3ecd5b3fd62f0b6b957086d4.jpg temp/1741326387_1298767_1341396818_109909176081710b7344d942043d59d5.jpg temp/1741326387_1298767_1341396806_7882f5b48ed2ca90a26b42e8347d78bd.jpg temp/1741326387_1298767_1341396717_54001f05bcf879afacb1a87d79058bc8.jpg temp/1741326387_1298767_1341396715_32a5a610fc8f9281ae19c25043998d4a.jpg temp/1741326387_1298767_1341396633_ea65c70a9f1b6fd14b9a6c6d27091a5b.jpg temp/1741326387_1298767_1341396627_4755826141267cd580c6a147ffa27e3b.jpg Found 94 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1341402794, 1341402791, 1341402761, 1341402756, 1341402570, 1341402566, 1341402542, 1341402538, 1341402473, 1341402468, 1341402412, 1341402408, 1341402300, 1341402288, 1341402196, 1341402193, 1341402143, 1341402140, 1341402108, 1341402101, 1341401885, 1341401882, 1341401771, 1341401768, 1341401664, 1341401662, 1341401304, 1341401302, 1341401162, 1341401160, 1341401072, 1341401067, 1341400985, 1341400966, 1341400604, 1341400600, 1341400522, 1341400518, 1341400385, 1341400382, 1341400337, 1341400332, 1341399934, 1341399930, 1341399864, 1341399858, 1341398981, 1341398977, 1341398911, 1341398897, 1341398742, 1341398732, 1341398683, 1341398679, 1341398622, 1341398616, 1341398515, 1341398509, 1341398365, 1341398347, 1341398197, 1341398193, 1341398146, 1341398140, 1341398077, 1341398073, 1341398052, 1341398050, 1341398025, 1341398021, 1341398013, 1341398011, 1341397869, 1341397865, 1341397840, 1341397836, 1341397795, 1341397792, 1341397769, 1341397765, 1341397709, 1341397706, 1341397523, 1341397517, 1341397322, 1341397259, 1341396868, 1341396863, 1341396818, 1341396806, 1341396717, 1341396715, 1341396633, 1341396627] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 94 time used for this insertion : 0.14303374290466309 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 36.7125289440155 time spend to save output : 0.1631169319152832 total time spend for step 0 : 36.875645875930786 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.02user 5.02system 0:54.16elapsed 33%CPU (0avgtext+0avgdata 2282812maxresident)k 734416inputs+204752outputs (856major+629077minor)pagefaults 0swaps