python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 205719 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 18240>100 please execute split_portfolio.py -i 22806757 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2809533'] with mtr_portfolio_ids : ['22806757'] and first list_photo_ids : [] new path : /proc/205719/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 21.761529684066772 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon Sep 1 17:40:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-09-01 17:40:54.885778: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-01 17:40:54.917196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-01 17:40:54.917515: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-01 17:40:54.919667: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-01 17:40:54.921706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-01 17:40:54.922138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-01 17:40:54.924527: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-01 17:40:54.925764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-01 17:40:54.930021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-01 17:40:54.931550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-01 17:40:54.931902: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-01 17:40:54.960549: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-01 17:40:54.962411: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fdbf8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-01 17:40:54.962462: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-01 17:40:55.071560: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x433767d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-01 17:40:55.071592: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-01 17:40:55.072378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-01 17:40:55.072454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-01 17:40:55.072474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-01 17:40:55.072491: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-01 17:40:55.072507: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-01 17:40:55.072523: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-01 17:40:55.072539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-01 17:40:55.072556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-01 17:40:55.073812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-01 17:40:55.073858: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-01 17:40:55.074602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-01 17:40:55.074617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-01 17:40:55.074626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-01 17:40:55.075919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 9799 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.237370252609253 time used to load_weights : 0.3851139545440674 0it [00:00, ?it/s] 42it [00:00, 410.04it/s] 84it [00:00, 387.35it/s] 100it [00:00, 382.68it/s]2025-09-01 17:41:07.588070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-01 17:41:07.593957: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-09-01 17:41:07.626524: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-01 17:41:07.630008: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-09-01 17:41:07.631624: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1756741231_205719_1361523518_b34db9ead043988d13cf807e97f8c6e5_60035060.jpg temp/1756741231_205719_1361523517_7c2da0c04ec62026d4f9970f2b23c005_46539132.jpg temp/1756741231_205719_1361523516_948c2c321700924644ed0a63c57fdeb2_61608453.jpg temp/1756741231_205719_1361523515_bbdd6f18850b79553fcbea75f9c8c1d3_72300146.jpg temp/1756741231_205719_1361523511_1c1665e1c8e1dbbcb031a6884f53b687_21726441.jpg temp/1756741231_205719_1361523510_002cf428a68d03abf695a5bf4510def6_89370789.jpg temp/1756741231_205719_1361523509_29c40c4cce9858ef67c53311a9c2a48d_77791095.jpg temp/1756741231_205719_1361523508_429e9e2b3b0bd56d868e404ee77e1f7e_18935968.jpg temp/1756741231_205719_1361523507_6463cd2e2d0b94b00be6000c20a76809_88384605.jpg temp/1756741231_205719_1361523506_c1dbc07b914fb7eb2c6d43847ee2943a_1840747.jpg temp/1756741231_205719_1361523504_ed81eed9754776021062a8337891ee43_49344063.jpg temp/1756741231_205719_1361523502_7b31ba3156dc6a76088c75017a045602_44026899.jpg temp/1756741231_205719_1361523501_2b0ba110fb4045f7dc9021143061396e_12613199.jpg temp/1756741231_205719_1361523500_0c666b9bdd9f7c85e36c3f21639a38cc_74292395.jpg temp/1756741231_205719_1361523499_27f46d9adcb80f517ba70be1c5d2c4ea_10884833.jpg temp/1756741231_205719_1361523498_306d5d83dd1ef179ced0ca291a20bfa1_56659644.jpg temp/1756741231_205719_1361523495_5f73e8de4b35a9997929a926e974c8b9_26073949.jpg temp/1756741231_205719_1361523494_0370d94934f7a7f8813bb6c9450610df_19348790.jpg temp/1756741231_205719_1361523493_88334ff417bdece1f67f68e5665393d6_84902622.jpg temp/1756741231_205719_1361523492_170fa50394cbd4b4882537dfbd547e23_96354217.jpg temp/1756741231_205719_1361523491_b5f873e2cc7ef0d892e545ff8980e6c9_40699117.jpg temp/1756741231_205719_1361523490_adefcb0ad50375971e27520a655fc4a4_74439596.jpg temp/1756741231_205719_1361523485_b282329ceed0faadb32b704363dd3c3b_89987447.jpg temp/1756741231_205719_1361523484_21de3b6553d8ced96f45a56556a24857_75823382.jpg temp/1756741231_205719_1361523483_b93af1b43d48efa93ea15b2a87f27ff5_17040058.jpg temp/1756741231_205719_1361523482_074685f5c7ea4d89e598e15cb85147df_52296394.jpg temp/1756741231_205719_1361523481_1bb904b418606fe2993f67b35544e6fe_64335009.jpg temp/1756741231_205719_1361523480_273762d877c6039454d76382ca496c3f_66428954.jpg temp/1756741231_205719_1361523474_9a290d78df97076c4749c3bfade76ef2_84433231.jpg temp/1756741231_205719_1361523473_a69792e97e2e2ae980898840fe820b17_25453188.jpg temp/1756741231_205719_1361523472_ab5324b4eedb4c0c27ed45d12b4f813a_25194998.jpg temp/1756741231_205719_1361523470_d53676be697943d3a82ac6764c206f98_28407161.jpg temp/1756741231_205719_1361523469_4a759219c4a0d21cc9fbc802f3bb5130_50586559.jpg temp/1756741231_205719_1361523467_621d819ec3985ffdb244c3862a8d6d1a_39184189.jpg temp/1756741231_205719_1361523458_a0f66b12d52eb21f693d8cfe0ea75cba_84669130.jpg temp/1756741231_205719_1361523457_01d4c8c187ca54ad97b5dff971910d93_85695197.jpg temp/1756741231_205719_1361523455_6e30c0be51e99c18930d16bfe2e7cf97_71060408.jpg temp/1756741231_205719_1361523454_2a72cd96d041623f75dc4edc7cc67b6b_19100501.jpg temp/1756741231_205719_1361523453_9681bb3312f43fc6de3775b075c70e3a_34317709.jpg temp/1756741231_205719_1361523452_7965f42411396a8e5e73816613ad2f88_38257296.jpg temp/1756741231_205719_1361523451_f27040fac7cb0daf4af85404fa178bf6_63356175.jpg temp/1756741231_205719_1361523450_7b4ecfc29cf51f727d13c82b6b4d070a_57748640.jpg temp/1756741231_205719_1361523449_9f2d6f98f2fe76b6a677e02cafc94800_7242710.jpg temp/1756741231_205719_1361523448_faec450ac8b5bac77c6b4239bb1c204f_87852883.jpg temp/1756741231_205719_1361523447_8d1fa23e702ab9fe9648119fea1eddc8_83311985.jpg temp/1756741231_205719_1361523446_255bb91d86062ffa6952af6a8aab4957_66486438.jpg temp/1756741231_205719_1361523444_ae24bee833ca504a7a8a82cf15cc67a4_44870921.jpg temp/1756741231_205719_1361523443_8e5b9d17170d767db93ebe14b533e3a7_14953143.jpg temp/1756741231_205719_1361523442_2a13daea37bb151ef2fd166824ef8cf9_76804869.jpg temp/1756741231_205719_1361523441_aa6160645dd298daa2cccbd512f4c334_96314139.jpg temp/1756741231_205719_1361523440_3d555e8f601fe2178a341f7148539aab_31123931.jpg temp/1756741231_205719_1361523439_4303cfd07238a0e0f19e6a06beb47bd5_69180207.jpg temp/1756741231_205719_1361523437_7f8026143a11b24a27ac928478ca033b_55611011.jpg temp/1756741231_205719_1361523436_5c8da14b21af6e9d6defb267a84c5bf5_86521015.jpg temp/1756741231_205719_1361523435_0af8ea4303ed2c31c7dfe348c4a50bb9_76837878.jpg temp/1756741231_205719_1361523434_072f56bc03fd549c137b85dd2059d051_32469522.jpg temp/1756741231_205719_1361523433_26207b374bdb3cc27cf68de588dded38_73599612.jpg temp/1756741231_205719_1361523432_955b4128faaeda2c9cabcf1587735538_93268504.jpg temp/1756741231_205719_1361523430_77b70399ce17a03438d08856ef3e55af_203902.jpg temp/1756741231_205719_1361523429_8183123170e09557aeb4774717ef3154_10866305.jpg temp/1756741231_205719_1361523428_53af40724e2ee5e2051f169680cb11d6_6853972.jpg temp/1756741231_205719_1361523427_2300136574071769e100e373fb146f59_55584756.jpg temp/1756741231_205719_1361523426_d83cc6031bf3b9998336439fabbfee43_97270460.jpg temp/1756741231_205719_1361523425_d531465101147bb5c87adce2f640aa0c_94394373.jpg temp/1756741231_205719_1361523424_8c140faa85a16fd26bfe517f63704672_49244371.jpg temp/1756741231_205719_1361523423_cb487d1efdf8da3b3ac8c2db887de179_87278623.jpg temp/1756741231_205719_1361523422_9022f87feb67c5acd96354fcb0a2f760_54911333.jpg temp/1756741231_205719_1361523421_bd5625af76faf953ba6bbf4adbe8f894_54186717.jpg temp/1756741231_205719_1361523420_885e41e6cf1da4480886c0a428a9366f_59554995.jpg temp/1756741231_205719_1361523419_ecf222c131428803b4a7c7b99defb2bf_84990641.jpg temp/1756741231_205719_1361523415_c057c914abca7895559a575f95e4db43_53003882.jpg temp/1756741231_205719_1361523414_7f921d067a646dbb71bca861b51f060f_655414.jpg temp/1756741231_205719_1361523413_07f356b83af8278bad02fb9058aafe79_37394087.jpg temp/1756741231_205719_1361523412_41de8ed5740e8b66bd4e8f11d4630fe6_67222245.jpg temp/1756741231_205719_1361523411_147fa429943c18e4c3f292602a113568_12019956.jpg temp/1756741231_205719_1361523410_8bdd626dd918449c7d140306c2d72ff7_69007059.jpg temp/1756741231_205719_1361523405_ea9096a1378b44613975817db836d996_44356353.jpg temp/1756741231_205719_1361523404_ed4a7ed11648dc6d969e11f5d4519d72_28382332.jpg temp/1756741231_205719_1361523403_7b52b85282d671b03ff382e7016e1a8b_5083164.jpg temp/1756741231_205719_1361523401_28474c354733b67c062184b478001260_82766140.jpg temp/1756741231_205719_1361523400_be679fff1231db428a1be13357aec8c3_84742035.jpg temp/1756741231_205719_1361523399_b61a42e0511b8cfad74e8a69af7a39fe_74837074.jpg temp/1756741231_205719_1361523397_ddcc4c18208a65b477436e5b8f45fb09_46016886.jpg temp/1756741231_205719_1361523396_b98300479dfd76f885e16e32ec7cee41_58934675.jpg temp/1756741231_205719_1361523395_2897312cdf64bca1d3944b209dff5629_83929779.jpg temp/1756741231_205719_1361523394_a9848b17676abfb0c59c2044e39c12e5_8865919.jpg temp/1756741231_205719_1361523393_a0494dcc1ef8c396a3375ce264805f40_15264812.jpg temp/1756741231_205719_1361523392_96855ef247b971bedcfbd858fc7060ff_72767496.jpg temp/1756741231_205719_1361523388_2dbd646714618f46d588ce5e46480dbe_14656815.jpg temp/1756741231_205719_1361523387_66d2af3d2d0032996be0bf1aa23355af_20466687.jpg temp/1756741231_205719_1361523386_476febadff8646751f14462bb38eca70_93206637.jpg temp/1756741231_205719_1361523385_8bd5d173b8c0e8a5c18335b4d684d449_46180319.jpg temp/1756741231_205719_1361523384_340d3c0d4667dc13480a99a2df3730ed_9316987.jpg temp/1756741231_205719_1361523383_e6d836e2cfd28d7226cb0d60d70122d8_53132915.jpg temp/1756741231_205719_1361523375_1d8dad1cbd95e38fd7d0cd755dcc945c_40971093.jpg temp/1756741231_205719_1361523373_ff8843ac6f2959accce9cf1ef18e2994_80863309.jpg temp/1756741231_205719_1361523372_fb9ce0381e3a0bd1e73a531fabdb9ec4_16118776.jpg temp/1756741231_205719_1361523371_fc57d7f394c9838ddd4fd070c58169a7_17720331.jpg temp/1756741231_205719_1361523370_1d0de0909e34d368c318f6d465903985_22025842.jpg temp/1756741231_205719_1361523369_190cb39678ab4d5407745c2de3259bae_23895388.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361523518, 1361523517, 1361523516, 1361523515, 1361523511, 1361523510, 1361523509, 1361523508, 1361523507, 1361523506, 1361523504, 1361523502, 1361523501, 1361523500, 1361523499, 1361523498, 1361523495, 1361523494, 1361523493, 1361523492, 1361523491, 1361523490, 1361523485, 1361523484, 1361523483, 1361523482, 1361523481, 1361523480, 1361523474, 1361523473, 1361523472, 1361523470, 1361523469, 1361523467, 1361523458, 1361523457, 1361523455, 1361523454, 1361523453, 1361523452, 1361523451, 1361523450, 1361523449, 1361523448, 1361523447, 1361523446, 1361523444, 1361523443, 1361523442, 1361523441, 1361523440, 1361523439, 1361523437, 1361523436, 1361523435, 1361523434, 1361523433, 1361523432, 1361523430, 1361523429, 1361523428, 1361523427, 1361523426, 1361523425, 1361523424, 1361523423, 1361523422, 1361523421, 1361523420, 1361523419, 1361523415, 1361523414, 1361523413, 1361523412, 1361523411, 1361523410, 1361523405, 1361523404, 1361523403, 1361523401, 1361523400, 1361523399, 1361523397, 1361523396, 1361523395, 1361523394, 1361523393, 1361523392, 1361523388, 1361523387, 1361523386, 1361523385, 1361523384, 1361523383, 1361523375, 1361523373, 1361523372, 1361523371, 1361523370, 1361523369] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.07489776611328125 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 15.527406454086304 time spend to save output : 0.08068323135375977 total time spend for step 0 : 15.608089685440063 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.44user 5.40system 0:43.66elapsed 45%CPU (0avgtext+0avgdata 1730108maxresident)k 855768inputs+175832outputs (2348major+564623minor)pagefaults 0swaps