python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 258209 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 546>100 please execute split_portfolio.py -i 23086561 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2886914'] with mtr_portfolio_ids : ['23086561'] and first list_photo_ids : [] new path : /proc/258209/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 12.344577074050903 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat May 17 12:52:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-05-17 12:52:47.239439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-17 12:52:47.272723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-17 12:52:47.273037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 12:52:47.275527: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 12:52:47.277510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-17 12:52:47.277850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-17 12:52:47.280846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-17 12:52:47.282553: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-17 12:52:47.288874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 12:52:47.290001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-17 12:52:47.290465: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-17 12:52:47.300408: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-17 12:52:47.302267: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f0550000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-17 12:52:47.302289: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-17 12:52:47.535408: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d44af50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-17 12:52:47.535462: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-17 12:52:47.536649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-17 12:52:47.536773: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 12:52:47.536809: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 12:52:47.536842: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-17 12:52:47.536872: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-17 12:52:47.536903: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-17 12:52:47.536933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-17 12:52:47.536965: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 12:52:47.538433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-17 12:52:47.538514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 12:52:47.539360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-17 12:52:47.539393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-17 12:52:47.539407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-17 12:52:47.540867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2940 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-17 12:52:54.288951: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-17 12:52:54.289606: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.07519006729126 time used to load_weights : 0.8524117469787598 0it [00:00, ?it/s] 100it [00:00, 1971.80it/s]2025-05-17 12:52:58.589198: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 12:52:58.822551: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 12:52:59.030570: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-17 12:52:59.038467: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1747479146_258209_1359092763_3547b4b2bc426a238da177ea52380db2.jpg temp/1747479146_258209_1359092760_69c895b4d3fda7e310dd3555b5f062fc.jpg temp/1747479146_258209_1359092740_79ceb16331f2cbc690ac66894d820efa.jpg temp/1747479146_258209_1359092737_7d3da27108c0065da6e807bb16a461c4.jpg temp/1747479146_258209_1359092719_56d6fbedd28987cc6be8d5b2cda0bdad.jpg temp/1747479146_258209_1359092716_cee643ae939a858272babe92e9392416.jpg temp/1747479146_258209_1359092695_70062f27568aaa7601d511f238b4a342.jpg temp/1747479146_258209_1359092676_e6df6d5be0907896565dacfcbb441565.jpg temp/1747479146_258209_1359092673_28baf60c977f322c02c8bf5c8929b760.jpg temp/1747479146_258209_1359092654_f65dcba14a118285fd34de344b3bfd15.jpg temp/1747479146_258209_1359092650_4a8262d50673f176a9ebd4b557ceb7b4.jpg temp/1747479146_258209_1359092632_b8648b34d9920f5e5602f1e96e62fcbd.jpg temp/1747479146_258209_1359092631_03cdc365871b7951f82dadd1168a1e98.jpg temp/1747479146_258209_1359092608_1d1c063193efd7cd742039dbd850a571.jpg temp/1747479146_258209_1359092605_4d923fdc1111a71b3957c7d2227e7404.jpg temp/1747479146_258209_1359092587_4481037f86233e829a8b1bc59ed654e9.jpg temp/1747479146_258209_1359092585_3e8b82c38446641d311a68bcd7032f3d.jpg temp/1747479146_258209_1359092563_1f71ad610f0189235206f47bca5f0ee7.jpg temp/1747479146_258209_1359092561_8451a623ca9c8042110750a2fb4647a9.jpg temp/1747479146_258209_1359092538_e5f80f4eb47fcc13f57ba51324cc65f0.jpg temp/1747479146_258209_1359092536_cfac988d90db2e0753ebe2711c2a43c8.jpg temp/1747479146_258209_1359092513_15683ba1d63cd6c72bb094d2810ecf34.jpg temp/1747479146_258209_1359092510_28e35589e5a1810d9b3f15ea75fe90ba.jpg temp/1747479146_258209_1359092490_4be678bbe842df372d38c1837f91f539.jpg temp/1747479146_258209_1359092487_bbcf447e1ecb03afddf65878cbc31955.jpg temp/1747479146_258209_1359092469_77447ea61b17af4120c8a14192a07f19.jpg temp/1747479146_258209_1359092466_7ecad57de6f6fcff5d243c541aa89b19.jpg temp/1747479146_258209_1359092445_2722ab0526008b38dbedd8f8cce1ba1e.jpg temp/1747479146_258209_1359092443_49d21f3624978f261aef13b6e4e65f3e.jpg temp/1747479146_258209_1359092374_62b3504010e5557d275d66b4e10b121f.jpg temp/1747479146_258209_1359092370_8be20e5c8c3c2a5b81f467eb509b3aeb.jpg temp/1747479146_258209_1359092348_e3d104ceb3dac2e5d12724089c0ba971.jpg temp/1747479146_258209_1359092345_25f8bbb85f54314ac7d9b4c4c1930fc4.jpg temp/1747479146_258209_1359092322_ee08c9bf3850b9f375d0792293b1bea4.jpg temp/1747479146_258209_1359092318_d2bbb8583fb326f9789c9081ba6d0077.jpg temp/1747479146_258209_1359092299_69b69d509fecf9a9b8172dceb16177e0.jpg temp/1747479146_258209_1359092297_9074c170c54c21eee36b2db777992c2d.jpg temp/1747479146_258209_1359092277_770f783372d35527bba09dff65b7c769.jpg temp/1747479146_258209_1359092274_6201af5f5a6f0fbad8258ea0153ef95f.jpg temp/1747479146_258209_1359092258_503cd02a1f942fbfc6238a5cf5aafbb4.jpg temp/1747479146_258209_1359092256_39fd50b897b14e402e079a649ad29abb.jpg temp/1747479146_258209_1359092234_a77b58b79ebdb9b84362260482fe8ea3.jpg temp/1747479146_258209_1359092232_c7de95234d9868f081064f89e77eecb8.jpg temp/1747479146_258209_1359092206_dc99af5f6c646f4bea93b36ef9793073.jpg temp/1747479146_258209_1359092204_46dcf0315f1847bc00dbe9aa84048512.jpg temp/1747479146_258209_1359092185_a042931ab4c0c2e8eb77484b8319169e.jpg temp/1747479146_258209_1359092182_5f2f17034b9196ca1df2db48f8b5203c.jpg temp/1747479146_258209_1359092163_ee67fbacb00e249c15cc208c09f1a352.jpg temp/1747479146_258209_1359092161_b31a13dc45f8ba8b4181db01d139e4ee.jpg temp/1747479146_258209_1359092148_2d6ebf2606a921d4f1013800bdb0e483.jpg temp/1747479146_258209_1359092146_b003f440185ef8ee421564331a3c38ab.jpg temp/1747479146_258209_1359092117_d46e7b6fd537a185c67a9ff8ce7cea3b.jpg temp/1747479146_258209_1359092115_145638f641f394de7e497229348a7ba6.jpg temp/1747479146_258209_1359092101_9c0f7aff959c81a91bc9620039bda961.jpg temp/1747479146_258209_1359092098_887e66ca9eb7118a0c18da8c64cef730.jpg temp/1747479146_258209_1359092078_aec156d613997013d782043932c31f01.jpg temp/1747479146_258209_1359092076_80a82938718ea325cbff6000bb9957f0.jpg temp/1747479146_258209_1359092055_70958336e0c7317f3c0a575b21d9530c.jpg temp/1747479146_258209_1359092053_aea6730b5c8a11bfda6ba3ed1e8130a3.jpg temp/1747479146_258209_1359092031_76738dd88caaf54fe9585878b731e262.jpg temp/1747479146_258209_1359092024_3c4ec0e90024f56ed80c7d48489434fd.jpg temp/1747479146_258209_1359092006_725ad7130c332bcd9b73e0d115d4c511.jpg temp/1747479146_258209_1359091983_8885868d36fd1b587f705b52aebb6398.jpg temp/1747479146_258209_1359091980_218735804fff401d64650460cc46e8a1.jpg temp/1747479146_258209_1359091743_07476842289912a101df8675463e3164.jpg temp/1747479146_258209_1359091725_e35287d9b1aa0ad3f268ce84a891ae08.jpg temp/1747479146_258209_1359091671_b196ee10c7192350f83e83e3a4b78604.jpg temp/1747479146_258209_1359091670_daa874189cbdebc1e23c8dc86be37e4b.jpg temp/1747479146_258209_1359091657_eaef09e6aa368d450d2c62f6a35dceb2.jpg temp/1747479146_258209_1359091635_b16ab8c9842cc2aa78a12ec2c51f3aeb.jpg temp/1747479146_258209_1359091632_b2c5d73aed02b53a0ef049a1ee9eef8e.jpg temp/1747479146_258209_1359091612_88fb66e27ac6571a7642505c828c2bd4.jpg temp/1747479146_258209_1359091609_268776d5da9debfa514bec6a4fab0737.jpg temp/1747479146_258209_1359091583_3a43aa25773956df4771ad832c36d633.jpg temp/1747479146_258209_1359091580_6cc87890f2b7e67090dd35474b49c66e.jpg temp/1747479146_258209_1359091555_188030c33c12c51016525fa05c37a5cc.jpg temp/1747479146_258209_1359091551_8f7a2c6c9bae858a508c3a72d0aa5fa5.jpg temp/1747479146_258209_1359091523_93653c8fef72a95c0c2824bf57d0cbb6.jpg temp/1747479146_258209_1359091519_cb1a42499cc0602a98c47e6c2a1e9d39.jpg temp/1747479146_258209_1359091502_e98dc01b598aa166fef2eb5d1eb863b8.jpg temp/1747479146_258209_1359091499_2d90018e862177700089918726675fb9.jpg temp/1747479146_258209_1359091474_6e18c2d3734c8aa41247bd9719f27264.jpg temp/1747479146_258209_1359091473_c3abe172f35f6e887abfbc84b1a9560e.jpg temp/1747479146_258209_1359091454_9bb3e9190a65155f715c3dd64c651057.jpg temp/1747479146_258209_1359091452_3ac595cf1554fd809caab2918393458d.jpg temp/1747479146_258209_1359091436_f75cf61ff1c145a8bb4353da017ef903.jpg temp/1747479146_258209_1359091434_83d28bc33696516d37d597a4454a634b.jpg temp/1747479146_258209_1359091409_8e7c661378204a28e1f76b048de9b607.jpg temp/1747479146_258209_1359091406_caecbbec172a72a9141e435fa731d95c.jpg temp/1747479146_258209_1359091382_9592aa2fe50cd24681a212576eda054c.jpg temp/1747479146_258209_1359091380_af56f4e3e2074eec473c169c054f809f.jpg temp/1747479146_258209_1359091360_3f3bbeac7d197aca3b69203673407ce7.jpg temp/1747479146_258209_1359091358_c931ed5c75c5b3768327c3cf728c0383.jpg temp/1747479146_258209_1359091337_5e672187bbdbcf04785a92d6c2851929.jpg temp/1747479146_258209_1359091327_5ed0529e148d71f272995c2e6a6506f6.jpg temp/1747479146_258209_1359091305_575eba9ffd45fc5bf6fee9d74b184e16.jpg temp/1747479146_258209_1359091302_56312a92058c98c17b3b20e4d7798b42.jpg temp/1747479146_258209_1359091228_d3f83addc866ade5ffa019a0bab1c9fc.jpg temp/1747479146_258209_1359091225_1867c631cd5b00adb50c844b85b6e43b.jpg temp/1747479146_258209_1359091164_c10b6d55cb8c18dac7acfff8fb429e64.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1359092763, 1359092760, 1359092740, 1359092737, 1359092719, 1359092716, 1359092695, 1359092676, 1359092673, 1359092654, 1359092650, 1359092632, 1359092631, 1359092608, 1359092605, 1359092587, 1359092585, 1359092563, 1359092561, 1359092538, 1359092536, 1359092513, 1359092510, 1359092490, 1359092487, 1359092469, 1359092466, 1359092445, 1359092443, 1359092374, 1359092370, 1359092348, 1359092345, 1359092322, 1359092318, 1359092299, 1359092297, 1359092277, 1359092274, 1359092258, 1359092256, 1359092234, 1359092232, 1359092206, 1359092204, 1359092185, 1359092182, 1359092163, 1359092161, 1359092148, 1359092146, 1359092117, 1359092115, 1359092101, 1359092098, 1359092078, 1359092076, 1359092055, 1359092053, 1359092031, 1359092024, 1359092006, 1359091983, 1359091980, 1359091743, 1359091725, 1359091671, 1359091670, 1359091657, 1359091635, 1359091632, 1359091612, 1359091609, 1359091583, 1359091580, 1359091555, 1359091551, 1359091523, 1359091519, 1359091502, 1359091499, 1359091474, 1359091473, 1359091454, 1359091452, 1359091436, 1359091434, 1359091409, 1359091406, 1359091382, 1359091380, 1359091360, 1359091358, 1359091337, 1359091327, 1359091305, 1359091302, 1359091228, 1359091225, 1359091164] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.05601668357849121 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 19.752718210220337 time spend to save output : 0.062310218811035156 total time spend for step 0 : 19.815028429031372 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.56user 4.90system 0:35.60elapsed 51%CPU (0avgtext+0avgdata 2040108maxresident)k 39448inputs+91048outputs (82major+642796minor)pagefaults 0swaps