python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1495577 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 13194>100 please execute split_portfolio.py -i 21033342 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2634615'] with mtr_portfolio_ids : ['21033342'] and first list_photo_ids : [] new path : /proc/1495577/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 16.48708939552307 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Tue Mar 25 12:46:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-03-25 12:46:51.874413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-25 12:46:51.903341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-25 12:46:51.903902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-25 12:46:51.907897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-25 12:46:51.910708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-25 12:46:51.911994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-25 12:46:51.915940: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-25 12:46:51.917567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-25 12:46:51.932967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-25 12:46:51.934333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-25 12:46:51.934982: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-25 12:46:51.963229: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-25 12:46:51.965779: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7efc24000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-25 12:46:51.965819: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-25 12:46:52.098658: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3bc3e690 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-25 12:46:52.098721: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-25 12:46:52.099787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-25 12:46:52.099880: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-25 12:46:52.099902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-25 12:46:52.099922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-25 12:46:52.099941: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-25 12:46:52.099960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-25 12:46:52.099981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-25 12:46:52.100002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-25 12:46:52.101254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-25 12:46:52.101319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-25 12:46:52.102155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-25 12:46:52.102181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-25 12:46:52.102205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-25 12:46:52.103626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2638 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-03-25 12:46:59.811940: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-25 12:46:59.812593: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-25 12:46:59.813196: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.669321775436401 time used to load_weights : 0.2714552879333496 0it [00:00, ?it/s] 100it [00:00, 2696.31it/s]2025-03-25 12:47:03.489457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-25 12:47:03.818414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-25 12:47:03.871004: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-03-25 12:47:03.875462: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1742903187_1495577_1347176342_8213616d580dea8fcfd060573b52f646.jpg temp/1742903187_1495577_1347176310_f120bf09b8c92523e5d92504f15d7c43.jpg temp/1742903187_1495577_1347176288_b0d943d337866684755851697df0fac9.jpg temp/1742903187_1495577_1347176264_0b37c577f6757968ce14ec180521d830.jpg temp/1742903187_1495577_1347176256_67ad46762e3a84af5dd0fd0dd11052b6.jpg temp/1742903187_1495577_1347176252_6cbac6d0e1f9e7f77b40a5e3efc4b596.jpg temp/1742903187_1495577_1347176209_824f6e8378023f23cf993fa0c2f7db3f.jpg temp/1742903187_1495577_1347176180_7b1fad88914c092ce66ec04ec446d978.jpg temp/1742903187_1495577_1347176156_fbad66558aff177d7ed3f2eb7be34595.jpg temp/1742903187_1495577_1347176129_75e04f8e355077ee81908fd75fa16483.jpg temp/1742903187_1495577_1347176106_131e26c1fa25f0456cac7bc43a08f5b0.jpg temp/1742903187_1495577_1347176057_11bbe6dbf4efbd907509189c4c1f6861.jpg temp/1742903187_1495577_1347175944_1d64311a84ca47f55e8ddf688caaf062.jpg temp/1742903187_1495577_1347175939_81c7fd9d889347a3f0c62b6e496f6b08.jpg temp/1742903187_1495577_1347175934_fd7e73571dac7707e07b9fcdf0fb7b46.jpg temp/1742903187_1495577_1347175926_003db7fb353e812507f88364b735b8f5.jpg temp/1742903187_1495577_1347175901_3594825afe44ccb36a5439d2cfe18cdd.jpg temp/1742903187_1495577_1347175784_0d34688f3119231d2edc5aee664789ea.jpg temp/1742903187_1495577_1347175687_6512441b51f6235eca1710c25a176107.jpg temp/1742903187_1495577_1347175652_62e15b8037ef81c03aeb3090a97c5b1e.jpg temp/1742903187_1495577_1347175628_d8d34779d539846fe62127d6bbf07a3c.jpg temp/1742903187_1495577_1347175604_41fb0de381e0e60fa566f130b018c62c.jpg temp/1742903187_1495577_1347175572_e2a84198e57ac8a90734a0fc3c73e177.jpg temp/1742903187_1495577_1347175550_558f21fc1a4d6cb000aded51ea7f405a.jpg temp/1742903187_1495577_1347175452_65edb29bb652e5226399469bba1027cb.jpg temp/1742903187_1495577_1347175438_7a31d33d9431f2aa00ccf4cfc9f1753e.jpg temp/1742903187_1495577_1347175410_e00da7c55ef92675a22e5a2face30d24.jpg temp/1742903187_1495577_1347175382_8e971a9a7e22688851e89903357698b8.jpg temp/1742903187_1495577_1347175365_cb088746f40ccb30aeca84fcc2183109.jpg temp/1742903187_1495577_1347175348_bdf1c2d64c1ec228faeece1921463a3b.jpg temp/1742903187_1495577_1347175031_686edc564dcc11dfc2c001875f6d229c.jpg temp/1742903187_1495577_1347175008_0743360327dc28a1eca361b7068bf511.jpg temp/1742903187_1495577_1347174974_f217572d14cbc83ddd0676f325ae2211.jpg temp/1742903187_1495577_1347174966_23339fdf77b44f7637b256f60869c1bc.jpg temp/1742903187_1495577_1347174947_90d3359e968bd9481a3e37780d70971f.jpg temp/1742903187_1495577_1347174920_54c525134caa9bbc31374812789dec49.jpg temp/1742903187_1495577_1347174687_19c5aa3939f6191023a5f526f12d5c97.jpg temp/1742903187_1495577_1347174655_16252318fa2c74514d8fe293d7ffbae8.jpg temp/1742903187_1495577_1347174640_3997765ddfbd49f6511faa309e200178.jpg temp/1742903187_1495577_1347174617_f999fd1b3f749a51cacbc6a5d2f9d278.jpg temp/1742903187_1495577_1347174576_cc2f3ef19a781b94319f8c245e0756b8.jpg temp/1742903187_1495577_1347174539_7ac09b42ab201b5e56c7729f50299419.jpg temp/1742903187_1495577_1347174487_979c49dc6b71c5654bb5374b6c356205.jpg temp/1742903187_1495577_1347174483_8dc3d6dc0b9d74ea36c93b78077cf989.jpg temp/1742903187_1495577_1347174449_d675b809f9697979900d9f98d52a391a.jpg temp/1742903187_1495577_1347174419_8bbdbb47454f30b68cea330874f3dfab.jpg temp/1742903187_1495577_1347174397_d77d1dfeb093ca3692c25ec23f5e1ae4.jpg temp/1742903187_1495577_1347174370_ffbb15820ebb9660aac17cd771078ecc.jpg temp/1742903187_1495577_1347174319_a81687391518646c832ff34c9decf299.jpg temp/1742903187_1495577_1347174310_1a28c39fd444c96da728b809871bd3c1.jpg temp/1742903187_1495577_1347174298_35a9847716d9c5bb8fcfc0856380db60.jpg temp/1742903187_1495577_1347174291_094f57d25a142f6f84c81dea36f984a3.jpg temp/1742903187_1495577_1347174265_b0759665d69e78d77d430a6d00b1550a.jpg temp/1742903187_1495577_1347174236_4274f2ddfb737f90b5ca871edcfce260.jpg temp/1742903187_1495577_1347174185_66f2d6cf67302cf0d10ff8124e2ba7e6.jpg temp/1742903187_1495577_1347174182_d32e82a37aaf440375f5455a763e06c8.jpg temp/1742903187_1495577_1347174178_71df057f0a16686543b8d775f8d71b83.jpg temp/1742903187_1495577_1347174172_b69fbd02ab9453c299eb9742562bdd8b.jpg temp/1742903187_1495577_1347174148_27d570cfd3b3e7c37dc920fc3507ed5c.jpg temp/1742903187_1495577_1347174122_0441f8958a6d74e73c933ff38f9db2d0.jpg temp/1742903187_1495577_1347174010_baa7aa0ff47fccc8fe9f34a9129be324.jpg temp/1742903187_1495577_1347173979_3e26fc08636382e54366891df2dce469.jpg temp/1742903187_1495577_1347173959_1c4c55bbc276bdde5f539ee655463311.jpg temp/1742903187_1495577_1347173931_cf31e40becc136bb810b9634dc3b1dac.jpg temp/1742903187_1495577_1347173904_4381becad9f7b373314cfaa4dfed0259.jpg temp/1742903187_1495577_1347173880_7ed40fc4a5123f5048ff30ea0c6ee2ea.jpg temp/1742903187_1495577_1347173781_c7aebc539192466a376c05aae9c2d0a4.jpg temp/1742903187_1495577_1347173759_4a58a41bcfb0d2021ed7762c45ce6a23.jpg temp/1742903187_1495577_1347173742_d93e178af467687c39f5399d5aa345d8.jpg temp/1742903187_1495577_1347173739_6bce1b71d19401c40ffeb1014f900d8e.jpg temp/1742903187_1495577_1347173736_15e0a0303196f224f49f1a2eb05c5063.jpg temp/1742903187_1495577_1347173732_3d14847631ef0c1d0d2f8d5ac27b2a28.jpg temp/1742903187_1495577_1347173614_51e5ea285337b7a9ca7be767ae01cad6.jpg temp/1742903187_1495577_1347173613_f81d04825b57598d31b0d6a06f983d76.jpg temp/1742903187_1495577_1347173610_d5b08a60103fcdbf27de6a2c7af06ce2.jpg temp/1742903187_1495577_1347173596_8a77fd66ce0f77b010499ff22b115f7e.jpg temp/1742903187_1495577_1347173568_8a00ba0f8eb89b354856deb0c7146a60.jpg temp/1742903187_1495577_1347173541_13bb17293fabd6c458557a58d5657019.jpg temp/1742903187_1495577_1347173431_7166af7dd57c0be5e4529be25c17f6d9.jpg temp/1742903187_1495577_1347173406_cdc7758e344330c91fffab8efe321c96.jpg temp/1742903187_1495577_1347173403_f9c48503e87b314e558b0e7fb03db76b.jpg temp/1742903187_1495577_1347173364_25dca1cedc2443dcf424fdc066a64054.jpg temp/1742903187_1495577_1347173361_a445127b92c292d0f12c554f5ec37bf1.jpg temp/1742903187_1495577_1347173358_f41d00595d822ef964859840af480008.jpg temp/1742903187_1495577_1347173334_16d0eaa54c113cc25a0ba3c141178633.jpg temp/1742903187_1495577_1347173332_da90f08c7ebf5026a03a006c51e85d8a.jpg temp/1742903187_1495577_1347173324_4e5f49596243ec3a21daf61a704e5e37.jpg temp/1742903187_1495577_1347173304_0213466943f2a54a6fd013c7b3f5b695.jpg temp/1742903187_1495577_1347173272_03281046f9c223c875097656b06599eb.jpg temp/1742903187_1495577_1347173252_0cc331cb6c1ff9e5b5f24ec17b3d5b1b.jpg temp/1742903187_1495577_1347173198_2f0fdb04fe1a80e41ab7336c6215cbcf.jpg temp/1742903187_1495577_1347173195_5add5e87cc788a717eb2f79942bd855f.jpg temp/1742903187_1495577_1347173187_f59466cbacb681b33da86249096177ef.jpg temp/1742903187_1495577_1347173184_87bb2f16d5b6dffc8ffb135f96043ef5.jpg temp/1742903187_1495577_1347173180_6abc0e37f3c28dd58f82f89bb43a02e2.jpg temp/1742903187_1495577_1347173167_2cb054e1248a7fbc634ef8d31a0f83dc.jpg temp/1742903187_1495577_1347173138_342816a416a422378889f1ee82fc694a.jpg temp/1742903187_1495577_1347173130_6b1bd9595ccecadac7169a8fbebb00a2.jpg temp/1742903187_1495577_1347173125_9c15dcedf0293b7d521b329e2e057d66.jpg temp/1742903187_1495577_1347173122_185e5e87cf6725db8f9a27861657a33e.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1347176342, 1347176310, 1347176288, 1347176264, 1347176256, 1347176252, 1347176209, 1347176180, 1347176156, 1347176129, 1347176106, 1347176057, 1347175944, 1347175939, 1347175934, 1347175926, 1347175901, 1347175784, 1347175687, 1347175652, 1347175628, 1347175604, 1347175572, 1347175550, 1347175452, 1347175438, 1347175410, 1347175382, 1347175365, 1347175348, 1347175031, 1347175008, 1347174974, 1347174966, 1347174947, 1347174920, 1347174687, 1347174655, 1347174640, 1347174617, 1347174576, 1347174539, 1347174487, 1347174483, 1347174449, 1347174419, 1347174397, 1347174370, 1347174319, 1347174310, 1347174298, 1347174291, 1347174265, 1347174236, 1347174185, 1347174182, 1347174178, 1347174172, 1347174148, 1347174122, 1347174010, 1347173979, 1347173959, 1347173931, 1347173904, 1347173880, 1347173781, 1347173759, 1347173742, 1347173739, 1347173736, 1347173732, 1347173614, 1347173613, 1347173610, 1347173596, 1347173568, 1347173541, 1347173431, 1347173406, 1347173403, 1347173364, 1347173361, 1347173358, 1347173334, 1347173332, 1347173324, 1347173304, 1347173272, 1347173252, 1347173198, 1347173195, 1347173187, 1347173184, 1347173180, 1347173167, 1347173138, 1347173130, 1347173125, 1347173122] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.05353426933288574 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 20.068222284317017 time spend to save output : 0.0619354248046875 total time spend for step 0 : 20.130157709121704 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.59user 4.55system 0:41.91elapsed 45%CPU (0avgtext+0avgdata 1847624maxresident)k 566768inputs+90224outputs (1281major+542697minor)pagefaults 0swaps