python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1785033 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2322>100 please execute split_portfolio.py -i 21370383 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2664534'] with mtr_portfolio_ids : ['21370383'] and first list_photo_ids : [] new path : /proc/1785033/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 11.34135627746582 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu Mar 13 06:46:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-03-13 06:46:51.089248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-13 06:46:51.117131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-13 06:46:51.125819: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-13 06:46:51.179852: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-13 06:46:51.213508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-13 06:46:51.214797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-13 06:46:51.249767: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-13 06:46:51.260449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-13 06:46:51.340676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-13 06:46:51.342106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-13 06:46:51.346588: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-13 06:46:51.379243: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-13 06:46:51.381435: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe05c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-13 06:46:51.381498: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-13 06:46:51.619222: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d51d1c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-13 06:46:51.619274: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-13 06:46:51.619980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-13 06:46:51.620088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-13 06:46:51.620111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-13 06:46:51.620132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-13 06:46:51.620152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-13 06:46:51.620172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-13 06:46:51.620192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-13 06:46:51.620212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-13 06:46:51.621245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-13 06:46:51.621302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-13 06:46:51.622013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-13 06:46:51.622033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-13 06:46:51.622045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-13 06:46:51.623233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3175 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-03-13 06:46:59.093956: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.224575996398926 time used to load_weights : 1.0165584087371826 0it [00:00, ?it/s] 100it [00:00, 2774.82it/s]2025-03-13 06:47:03.691030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-13 06:47:03.892670: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-13 06:47:03.933140: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-03-13 06:47:03.941008: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1741844792_1785033_1343712988_3f7f74090a62a3f38aa26a1aa722abcb.jpg temp/1741844792_1785033_1343712982_78dd37e4572040ef26506bd0192e40d0.jpg temp/1741844792_1785033_1343712978_2af3860b74cb42d9e2c0a1125af50d4b.jpg temp/1741844792_1785033_1343712973_5bf2dcc1f4ea776f0a148ccb6b284fe8.jpg temp/1741844792_1785033_1343712968_474869322a1e4bcf9c6dfc5063e80f17.jpg temp/1741844792_1785033_1343712963_9d4d735e12421cc90ac3075bdd8d35ef.jpg temp/1741844792_1785033_1343712945_26c613dae41c0eb214e8dba76de67590.jpg temp/1741844792_1785033_1343712940_e188ec6421a289e1d8581e3c5609ecb1.jpg temp/1741844792_1785033_1343712934_917d998cae330a6b8f86636d12c696fa.jpg temp/1741844792_1785033_1343712927_e1da36a7e1cecb2efb5b226e12b2396d.jpg temp/1741844792_1785033_1343712923_dfaaa4f5a40909ebaaaafa67e8d44e3c.jpg temp/1741844792_1785033_1343712919_4303bc02a635efd39e025e653ba05573.jpg temp/1741844792_1785033_1343712904_292f51b3fa4df89b41ef42325c282ae2.jpg temp/1741844792_1785033_1343712900_2a590636a334ca7186cbec7db09c8f15.jpg temp/1741844792_1785033_1343712895_4deab35941d27f3075f231dc7d908be0.jpg temp/1741844792_1785033_1343712890_7899d3b0c41abe92a8c29570466980f2.jpg temp/1741844792_1785033_1343712886_cba6d786ddd6a2a5f4fdd0978e1128b6.jpg temp/1741844792_1785033_1343712882_d319404de449ef42d3cefbb3274fbbcf.jpg temp/1741844792_1785033_1343712865_4dbf34647e2bc224248364447f029486.jpg temp/1741844792_1785033_1343712861_2ddec2f05d3e2b16c7479aa1bec15160.jpg temp/1741844792_1785033_1343712856_9ef65f80246ce986114b16ac86dca9cc.jpg temp/1741844792_1785033_1343712851_2038d7401f071f8d9aea5a46b9950aa2.jpg temp/1741844792_1785033_1343712847_a7966b0f26c521742581c265ca4324e2.jpg temp/1741844792_1785033_1343712843_33f0e2469683806264b77f0d7e1faaa5.jpg temp/1741844792_1785033_1343712827_9d0ca592e15990b4b25022d249380847.jpg temp/1741844792_1785033_1343712823_d57517cb904650441393b90ec22bc1a0.jpg temp/1741844792_1785033_1343712819_a6879ab66f0e10ca399af9589cdb5e28.jpg temp/1741844792_1785033_1343712813_0e774c011b3fbdfb839dd69ef8e5688c.jpg temp/1741844792_1785033_1343712809_0dc8924127022f6ede9d26e2119dad7b.jpg temp/1741844792_1785033_1343712807_776bbb353895280c0b2c02f1af588354.jpg temp/1741844792_1785033_1343712781_e3ce9a42c62f8c9fc674d8388ea3e5cc.jpg temp/1741844792_1785033_1343712777_e1e0754bd7981e8b2702f6bab1984bba.jpg temp/1741844792_1785033_1343712773_6623d287dbabc9704790b7d6a0970221.jpg temp/1741844792_1785033_1343712768_b0240df169280e0dd5b67879ba0c3ada.jpg temp/1741844792_1785033_1343712762_0c0f6b68eb6d43f2ca7fd7e5412b160f.jpg temp/1741844792_1785033_1343712759_d577c7e6c8bceb6e531a69f3ee512651.jpg temp/1741844792_1785033_1343712738_86476303d9927dd8eb645105347eadc9.jpg temp/1741844792_1785033_1343712735_480acb23b74238f64e11420d9a1e273b.jpg temp/1741844792_1785033_1343712731_5bde9e383be4aa98e7c13d52ee65acbd.jpg temp/1741844792_1785033_1343712727_3d8204a3a50def7c6c489f5411f412e7.jpg temp/1741844792_1785033_1343712723_d12be29b70e67661e73e282b70771fe5.jpg temp/1741844792_1785033_1343712721_c8102e9c9338ec545f97aef2b03e388e.jpg temp/1741844792_1785033_1343712699_edf4ed92c66198cac916bf70cc3172c0.jpg temp/1741844792_1785033_1343712696_90900a51b2ae827cd23d779e2fd439e5.jpg temp/1741844792_1785033_1343712692_2084223d8ae1de3ea78744a730395153.jpg temp/1741844792_1785033_1343712687_dfd8c3085b550401de86b532935ca771.jpg temp/1741844792_1785033_1343712680_619819daa7204a9eb7bace5d67a65b63.jpg temp/1741844792_1785033_1343712678_d6bbfa77c6d540e36c717eaeadaa8086.jpg temp/1741844792_1785033_1343712658_6d1cbd944c90636afd66ffd83a5c7ab1.jpg temp/1741844792_1785033_1343712655_c3cae8560f2b32d7b9175abc1913478b.jpg temp/1741844792_1785033_1343712649_e1212adef0e7553225fa5dd33c7aca7f.jpg temp/1741844792_1785033_1343712644_ec02772190ba6ac83ea20b4f7b277d11.jpg temp/1741844792_1785033_1343712641_b5c9f5f612f714271a4093b9e14e2ac0.jpg temp/1741844792_1785033_1343712637_024a8281c4e98a175a23da37944c9949.jpg temp/1741844792_1785033_1343712617_1d5e6637bf1c43f5dd49d1630f921120.jpg temp/1741844792_1785033_1343712615_282941a26bb04a3b91e1564cce8f3f75.jpg temp/1741844792_1785033_1343712610_bb8c7b56f91f0aba1809228bbf134289.jpg temp/1741844792_1785033_1343712605_3fe9ca6b9cc20365af9f8ad3c6c92d38.jpg temp/1741844792_1785033_1343712602_7d627c20798e38ac064fd1f61a64161b.jpg temp/1741844792_1785033_1343712600_61fb26bccdf752ef3ff400d850c2159e.jpg temp/1741844792_1785033_1343712579_93beeb0a3bf387c29724f196a82182b7.jpg temp/1741844792_1785033_1343712577_6cc9df1f2361e438f9f5d584b71b3674.jpg temp/1741844792_1785033_1343712574_7c3f5289c5ee2ae9ddb3695f468244c5.jpg temp/1741844792_1785033_1343712569_f311ff92912173105b90c8e9e6498c93.jpg temp/1741844792_1785033_1343712565_c8ef424c806146c66bf6023023c70406.jpg temp/1741844792_1785033_1343712562_47d5038edc96d952e217cff288175903.jpg temp/1741844792_1785033_1343712543_58fbd3883e78b0a054c681cc0a7938e1.jpg temp/1741844792_1785033_1343712540_573cfe6c75e65e1fcd275dea07efb743.jpg temp/1741844792_1785033_1343712536_8241c1bdd0d4365223ba92371db34236.jpg temp/1741844792_1785033_1343712531_d93c74ef73094a8530ee2d47e6fc0296.jpg temp/1741844792_1785033_1343712527_706c72dc7dd132c996a0233b43bfd1e5.jpg temp/1741844792_1785033_1343712525_48fce3e580a7c0a4f893b6be3cdab9d0.jpg temp/1741844792_1785033_1343712511_18d4588e3ffca5f6128a3584fff2bc92.jpg temp/1741844792_1785033_1343712507_35bea1ae193aae3b623eaccbcd1ef783.jpg temp/1741844792_1785033_1343712502_593a34de22aa584d80e354accdbeb587.jpg temp/1741844792_1785033_1343712499_7c3335b54a089f0b9538e0f0a09e971e.jpg temp/1741844792_1785033_1343712496_28619acd3fff742e6e001cd3ab499d9a.jpg temp/1741844792_1785033_1343712491_f3b20e6295dbc350ab986d7edd180d66.jpg temp/1741844792_1785033_1343712477_e41c70bbe7a39f2421afa514c6ad9f9f.jpg temp/1741844792_1785033_1343712472_e59c47c7b4ede7747a33dc1725fe60c7.jpg temp/1741844792_1785033_1343712467_215b880237156cd846bd6fe3f710e382.jpg temp/1741844792_1785033_1343712462_1681842f020b18142ccb81e0c5a4d5cf.jpg temp/1741844792_1785033_1343712459_43a8b1dce29ff98b3cb970717cbd747b.jpg temp/1741844792_1785033_1343712451_34a22042d05a998a44e6838749925398.jpg temp/1741844792_1785033_1343712440_07b7fab9f319fe503ed8e6872105add3.jpg temp/1741844792_1785033_1343712434_b50aa8b9cd83b741501027ba0fda52cd.jpg temp/1741844792_1785033_1343712433_04abd73078b0919e7ffe7547a97a00ea.jpg temp/1741844792_1785033_1343712428_14439201ca90d421932a0da67a96b954.jpg temp/1741844792_1785033_1343712423_a5691cc6f24c3f4fabb1c18aaf3ff258.jpg temp/1741844792_1785033_1343712416_d8a941f8bc0072563c66f5c9b66230df.jpg temp/1741844792_1785033_1343712403_b977a787d4e1165bcef6ba511acaedb3.jpg temp/1741844792_1785033_1343712400_c8f658ce07a9551ef516982b9248d818.jpg temp/1741844792_1785033_1343712397_93567ec725cb84fbbff317cb8786e135.jpg temp/1741844792_1785033_1343712393_095503085c01b5ead7c3d0d3736bf4cb.jpg temp/1741844792_1785033_1343712389_1760e847e3197a93d22ae0a2c112a868.jpg temp/1741844792_1785033_1343712384_eb67d7dcf17d6f36424c749bdd41c3ad.jpg temp/1741844792_1785033_1343712370_8d351955710f16233eff79b5c2254783.jpg temp/1741844792_1785033_1343712367_5646a9592c9ade3a99279ee57bdcbc31.jpg temp/1741844792_1785033_1343712363_9742212546f2be08854e0e870cefb640.jpg temp/1741844792_1785033_1343712359_8b1ad7eaa81462d282427d916ebb649d.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1343712988, 1343712982, 1343712978, 1343712973, 1343712968, 1343712963, 1343712945, 1343712940, 1343712934, 1343712927, 1343712923, 1343712919, 1343712904, 1343712900, 1343712895, 1343712890, 1343712886, 1343712882, 1343712865, 1343712861, 1343712856, 1343712851, 1343712847, 1343712843, 1343712827, 1343712823, 1343712819, 1343712813, 1343712809, 1343712807, 1343712781, 1343712777, 1343712773, 1343712768, 1343712762, 1343712759, 1343712738, 1343712735, 1343712731, 1343712727, 1343712723, 1343712721, 1343712699, 1343712696, 1343712692, 1343712687, 1343712680, 1343712678, 1343712658, 1343712655, 1343712649, 1343712644, 1343712641, 1343712637, 1343712617, 1343712615, 1343712610, 1343712605, 1343712602, 1343712600, 1343712579, 1343712577, 1343712574, 1343712569, 1343712565, 1343712562, 1343712543, 1343712540, 1343712536, 1343712531, 1343712527, 1343712525, 1343712511, 1343712507, 1343712502, 1343712499, 1343712496, 1343712491, 1343712477, 1343712472, 1343712467, 1343712462, 1343712459, 1343712451, 1343712440, 1343712434, 1343712433, 1343712428, 1343712423, 1343712416, 1343712403, 1343712400, 1343712397, 1343712393, 1343712389, 1343712384, 1343712370, 1343712367, 1343712363, 1343712359] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.05608558654785156 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 20.46788263320923 time spend to save output : 0.061799049377441406 total time spend for step 0 : 20.52968168258667 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 12.98user 5.34system 0:41.16elapsed 44%CPU (0avgtext+0avgdata 1879820maxresident)k 1033248inputs+105880outputs (4089major+643483minor)pagefaults 0swaps