python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1037491 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 15636>100 please execute split_portfolio.py -i 22867180 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2818857'] with mtr_portfolio_ids : ['22867180'] and first list_photo_ids : [] new path : /proc/1037491/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 13.526195049285889 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Wed Sep 3 05:28:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-09-03 05:28:49.053485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-03 05:28:49.084589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-03 05:28:49.085221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-03 05:28:49.088518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-03 05:28:49.091031: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-03 05:28:49.092206: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-03 05:28:49.094996: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-03 05:28:49.096065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-03 05:28:49.101468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-03 05:28:49.102559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-03 05:28:49.103084: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-03 05:28:49.128490: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-03 05:28:49.130746: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f366c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-03 05:28:49.130801: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-03 05:28:49.250531: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x394d9c40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-03 05:28:49.250575: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-03 05:28:49.251709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-03 05:28:49.251808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-03 05:28:49.251838: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-03 05:28:49.251865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-03 05:28:49.251892: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-03 05:28:49.251919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-03 05:28:49.251946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-03 05:28:49.251973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-03 05:28:49.253256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-03 05:28:49.253323: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-03 05:28:49.254139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-03 05:28:49.254162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-03 05:28:49.254176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-03 05:28:49.255510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2948 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-09-03 05:28:55.734325: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-03 05:28:55.734918: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.562283992767334 time used to load_weights : 0.26700830459594727 0it [00:00, ?it/s] 100it [00:00, 1450.68it/s]2025-09-03 05:28:59.131445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-03 05:28:59.364191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-03 05:28:59.645620: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-09-03 05:28:59.647340: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1756870107_1037491_1381021231_0a0dd7c570d475e4ddfbc85bfe091fe5_76662547.jpg temp/1756870107_1037491_1381021230_97aa547c10f5fbf399fe0873bd44d50c_91075619.jpg temp/1756870107_1037491_1381021229_a0f2b7eda88465033227a2e723302168_33172739.jpg temp/1756870107_1037491_1381021228_e015b1e148a3db52a34b1ca7ab5d712c_75710307.jpg temp/1756870107_1037491_1381021227_a763b0ecfaec9f6c738565f22ad6aead_12160588.jpg temp/1756870107_1037491_1381021226_476cbcb244aa6e08459e162a6c38f059_40909907.jpg temp/1756870107_1037491_1381021220_0255b3bbd19809a73db8e9a0dc0d8945_22874017.jpg temp/1756870107_1037491_1381021218_da406e090b9b7efa919e6b7f4090d513_19694963.jpg temp/1756870107_1037491_1381021217_e061bfefddff7576138c414e3344b2ab_71857568.jpg temp/1756870107_1037491_1381021216_6c91d7d851d423e7be67e4ff6ca2f1e6_97561014.jpg temp/1756870107_1037491_1381021215_e829004a977f51dc813f649d13380bf3_76906848.jpg temp/1756870107_1037491_1381021214_457272ef7090e61d6d3d8b463633564e_11505557.jpg temp/1756870107_1037491_1381021180_c86210069f2526a8194bc9f60a9a6ca5_58451111.jpg temp/1756870107_1037491_1381021179_d2d080abfad69eabf8cd40768cd1fafd_28798232.jpg temp/1756870107_1037491_1381021178_23eff991dea48de9afca5258f38ff8d2_77251152.jpg temp/1756870107_1037491_1381021177_3e32433a6189a2bba3101b27a6428f5b_43078261.jpg temp/1756870107_1037491_1381021176_6832f525fb9555159106b0040ad48ffd_21985361.jpg temp/1756870107_1037491_1381021175_23d4009936a0a0d0fbbc99a9e5dfa93b_4759790.jpg temp/1756870107_1037491_1381021166_bec5c0e4e88155093bd969cd4c0386a4_78186016.jpg temp/1756870107_1037491_1381021165_c757c2b1c5683672c86d8b925c902c5b_3229045.jpg temp/1756870107_1037491_1381021164_52b02c9bb22ee2949e7c7b468f98d265_85649805.jpg temp/1756870107_1037491_1381021163_07a8ee5c233c872fa7021ae95e0be68b_39929000.jpg temp/1756870107_1037491_1381021162_196ce543afbb22ea09e8739554e91e20_97704849.jpg temp/1756870107_1037491_1381021161_43dccb303f67acf4d737c64d3473e9c7_37761395.jpg temp/1756870107_1037491_1381021129_347a5ff36add416170c7ea76d6a4acc0_14803274.jpg temp/1756870107_1037491_1381021128_6d42a7bff333cd6b8c971f4aec8c00fc_24189128.jpg temp/1756870107_1037491_1381021127_61d84328480bc346e7e6d77bf4cdff09_67181119.jpg temp/1756870107_1037491_1381021126_4dd7c194e2d6222ed1c8d20159cc048d_91753862.jpg temp/1756870107_1037491_1381021125_e71c831eb5ac9482eaec709aae60c935_74628405.jpg temp/1756870107_1037491_1381021124_0cda4b53f4f1bcf556dbc48b075e26e8_3227489.jpg temp/1756870107_1037491_1381021108_5e61e0cd71ebfbbc24c5141f60429e69_6018731.jpg temp/1756870107_1037491_1381021107_069e4cbbd72e836b9c6ec0e96dbbb85d_81466115.jpg temp/1756870107_1037491_1381021106_326f35fd029186f1fe2f26bb374f003d_26948751.jpg temp/1756870107_1037491_1381021105_43baa42021a3318542411fea5ed07c29_17228195.jpg temp/1756870107_1037491_1381021104_60be10d4c58e89baed5313f6f55c0c75_62287496.jpg temp/1756870107_1037491_1381021103_854527347eac4ed4ce74176f2c7e9c25_71700763.jpg temp/1756870107_1037491_1381021101_8ad08e389db24f4630aa2e25dea07f1f_42273447.jpg temp/1756870107_1037491_1381021100_969fdad2a4e7bd468e7b9e95699d0c77_59774718.jpg temp/1756870107_1037491_1381021099_18d3f4fb6bace9596c0a0034a27096ff_95703809.jpg temp/1756870107_1037491_1381021097_d08c89ab4f628bf51c092e8d7d6e90f0_12086025.jpg temp/1756870107_1037491_1381021096_233df06ee801b76b2105ed5943c0fc09_11303184.jpg temp/1756870107_1037491_1381021095_80dd2b0ed355e980e88b7e1fd33c1bd7_21299672.jpg temp/1756870107_1037491_1381021060_c18c267d3f59c3b90694142bb0c9736b_10008713.jpg temp/1756870107_1037491_1381021059_3ae272457bb0efabb403f689eb9856ad_48743474.jpg temp/1756870107_1037491_1381021058_d01cc8c1d6b7797576b56e6401e657a3_8517462.jpg temp/1756870107_1037491_1381021057_089a9837a53b72b9602000e48e73a579_37953699.jpg temp/1756870107_1037491_1381021056_b8b2f668012bb639d0cc97f1e0e0ca52_84823515.jpg temp/1756870107_1037491_1381021055_94f466f076ccc753093ba1d8f7b8a2e4_13408319.jpg temp/1756870107_1037491_1381021047_607143a0831774e84f7ac4076c8fec40_39659856.jpg temp/1756870107_1037491_1381021046_9906f82b6264825d06bef9dc9e276f90_22891455.jpg temp/1756870107_1037491_1381021045_0a909ca0889781125444d589c5d1f5bc_53112105.jpg temp/1756870107_1037491_1381021044_937bda745fa5243b4ed4a0271e0fde04_35514198.jpg temp/1756870107_1037491_1381021043_586a64703415951848cc7848387e7180_65710421.jpg temp/1756870107_1037491_1381021042_71c81f0e98d00fbaeb7284b4db888560_319687.jpg temp/1756870107_1037491_1381021031_28ca895f16f740110bf5c43a90cf4f5b_65604835.jpg temp/1756870107_1037491_1381021030_51bd3bbb06fe7033d4a4cf794b2fa757_28020667.jpg temp/1756870107_1037491_1381021029_bd24fde5ad4480bc6dd6bfe70fd9f9bb_35279410.jpg temp/1756870107_1037491_1381021028_caa605937a5d69f79a618bf841e9ab72_59809284.jpg temp/1756870107_1037491_1381021027_0031ae90429a6af10c9e8c9b74f9dc43_56717601.jpg temp/1756870107_1037491_1381021026_ee16ba972e2b94841ff01946dfb8d108_98261743.jpg temp/1756870107_1037491_1381021013_59e25d2fe2bae2c91ace1c8c51ee8ae3_83639757.jpg temp/1756870107_1037491_1381021011_7231d027d950b54a35e35924d55aafc1_70946119.jpg temp/1756870107_1037491_1381021009_a62c3a6e9e146066fa9eb40ea86831a7_84897146.jpg temp/1756870107_1037491_1381021007_1c932fa501b7a8377bacb1209a12a977_28895674.jpg temp/1756870107_1037491_1381021005_ee7d29d4a88ad8e141d5e08d565525dd_69567156.jpg temp/1756870107_1037491_1381021003_9aa4642a97b62b2666ebfd592cbb70f9_37175427.jpg temp/1756870107_1037491_1381020992_9eb8a979cfae6b1a4ba6847cddde8bbe_39345309.jpg temp/1756870107_1037491_1381020991_4e524c11c527ac4c4778e2636b565a6a_72051203.jpg temp/1756870107_1037491_1381020990_a4e37c8a8f77fddab88732d68b936cd8_16167705.jpg temp/1756870107_1037491_1381020989_db2e123ff6ac6e21339e2a983901ccd3_77710310.jpg temp/1756870107_1037491_1381020988_74e595904206b240501b7835c4d3289a_59177153.jpg temp/1756870107_1037491_1381020987_49e80a18994a7a0661a334cdf59c298d_78638194.jpg temp/1756870107_1037491_1381020912_8746acbb0ad66adea279a2d3ceccca2a_56692786.jpg temp/1756870107_1037491_1381020910_406b456aeb80430de28dfcf17c7f6e2a_88700703.jpg temp/1756870107_1037491_1381020909_497f8b0016e28f22bfff656e3b595d65_47969485.jpg temp/1756870107_1037491_1381020908_e555e11d56150840542fe091ca978487_18237212.jpg temp/1756870107_1037491_1381020907_517e4c0393a44ae0a153463b6aab515f_31130215.jpg temp/1756870107_1037491_1381020906_ac2388ac66a0b68429672e0717cdbb03_27415941.jpg temp/1756870107_1037491_1381020892_4b4483cebbe3ce991cbc84f2896d56eb_41233814.jpg temp/1756870107_1037491_1381020891_af61a401dec9ad98682ac49c4d426be7_5150502.jpg temp/1756870107_1037491_1381020890_31d4b4f220e6cd4da498a4e8a1e3bab8_48513168.jpg temp/1756870107_1037491_1381020889_b7640a291527776cf8f0d2e933112815_14675523.jpg temp/1756870107_1037491_1381020888_3c21156eb70aa7aed5dab4cf5c09cbfa_31735947.jpg temp/1756870107_1037491_1381020887_b01c18229e77f42f3b2e70ffda8fc7b2_12867280.jpg temp/1756870107_1037491_1381020880_50d7ba73e10a54f1d3f253abb5c3a6ae_645513.jpg temp/1756870107_1037491_1381020879_7e75e6427bbe2315d249c2041e043785_81782309.jpg temp/1756870107_1037491_1381020878_dd430fc007e3b56a337a6e8e756abb61_15850496.jpg temp/1756870107_1037491_1381020877_fe214ffab8455aadcabd7d0290145297_72848057.jpg temp/1756870107_1037491_1381020876_f79582bf03d3e45eb9d0ede1242bc404_83592958.jpg temp/1756870107_1037491_1381020875_c9df3aba664882c4133a5bf0135d99e7_4979976.jpg temp/1756870107_1037491_1381020856_b23e2eb8cb0bc6a954ae7fe57bd3aec3_72774897.jpg temp/1756870107_1037491_1381020854_bbcc6cd629590fc607f11819385cae46_98867188.jpg temp/1756870107_1037491_1381020852_dbe2409015fd1d171df20e6a1d235e4d_8056170.jpg temp/1756870107_1037491_1381020850_bd24f49a634a9b6ac7daf413145e2051_51828262.jpg temp/1756870107_1037491_1381020849_8f7a00ed0db9f6ea7a1f45d6e7bce6d2_68753425.jpg temp/1756870107_1037491_1381020848_260e564bdc133ff67ca77a27d43d4faa_28292186.jpg temp/1756870107_1037491_1381020845_3e9c61be1c7b6a8917a43ec3c45f32cc_72374396.jpg temp/1756870107_1037491_1381020844_b69f004cf324f89df684aec7ba7df457_20051880.jpg temp/1756870107_1037491_1381020843_0197c25cb66f80f9eb37c8deb4b51a13_20889440.jpg temp/1756870107_1037491_1381020842_f126b1e54f164b23311a4a4e597900e0_28430994.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1381021231, 1381021230, 1381021229, 1381021228, 1381021227, 1381021226, 1381021220, 1381021218, 1381021217, 1381021216, 1381021215, 1381021214, 1381021180, 1381021179, 1381021178, 1381021177, 1381021176, 1381021175, 1381021166, 1381021165, 1381021164, 1381021163, 1381021162, 1381021161, 1381021129, 1381021128, 1381021127, 1381021126, 1381021125, 1381021124, 1381021108, 1381021107, 1381021106, 1381021105, 1381021104, 1381021103, 1381021101, 1381021100, 1381021099, 1381021097, 1381021096, 1381021095, 1381021060, 1381021059, 1381021058, 1381021057, 1381021056, 1381021055, 1381021047, 1381021046, 1381021045, 1381021044, 1381021043, 1381021042, 1381021031, 1381021030, 1381021029, 1381021028, 1381021027, 1381021026, 1381021013, 1381021011, 1381021009, 1381021007, 1381021005, 1381021003, 1381020992, 1381020991, 1381020990, 1381020989, 1381020988, 1381020987, 1381020912, 1381020910, 1381020909, 1381020908, 1381020907, 1381020906, 1381020892, 1381020891, 1381020890, 1381020889, 1381020888, 1381020887, 1381020880, 1381020879, 1381020878, 1381020877, 1381020876, 1381020875, 1381020856, 1381020854, 1381020852, 1381020850, 1381020849, 1381020848, 1381020845, 1381020844, 1381020843, 1381020842] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.07564783096313477 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 19.070989847183228 time spend to save output : 0.07897186279296875 total time spend for step 0 : 19.149961709976196 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 12.95user 4.44system 0:35.99elapsed 48%CPU (0avgtext+0avgdata 2050000maxresident)k 157440inputs+175392outputs (451major+588925minor)pagefaults 0swaps