python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2494884 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 17815>100 please execute split_portfolio.py -i 22792232 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2977394'] with mtr_portfolio_ids : ['22792232'] and first list_photo_ids : [] new path : /proc/2494884/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 15.564907789230347 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Wed May 28 18:31:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-05-28 18:31:50.102363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-28 18:31:50.132277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-28 18:31:50.132551: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 18:31:50.134409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 18:31:50.136321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-28 18:31:50.136693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-28 18:31:50.139462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-28 18:31:50.140877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-28 18:31:50.145675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 18:31:50.146869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-28 18:31:50.147321: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-28 18:31:50.175182: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-28 18:31:50.177324: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1558000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-28 18:31:50.177387: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-28 18:31:50.413619: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xa794900 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-28 18:31:50.413670: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-28 18:31:50.414606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-28 18:31:50.414714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 18:31:50.414750: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 18:31:50.414777: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-28 18:31:50.414798: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-28 18:31:50.414818: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-28 18:31:50.414837: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-28 18:31:50.414856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 18:31:50.416117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-28 18:31:50.416172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 18:31:50.416929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-28 18:31:50.416962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-28 18:31:50.416979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-28 18:31:50.418311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2695 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-28 18:31:57.235313: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-28 18:31:57.235985: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.125607252120972 time used to load_weights : 0.9582653045654297 0it [00:00, ?it/s] 100it [00:00, 1568.79it/s]2025-05-28 18:32:01.532174: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 18:32:01.749590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 18:32:01.871973: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-28 18:32:01.889064: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748449887_2494884_1361521587_d170a076e417a9fdaf38136a6e3f579f_81193362.jpg temp/1748449887_2494884_1361521585_1369fbcb73bceca34f3f030492b9c9f2_46090192.jpg temp/1748449887_2494884_1361521583_b3ff6a0f1dcb3268093680d67a302dc6_32043483.jpg temp/1748449887_2494884_1361521581_471d001f66651f7b36e95d6a1a861b4b_48280892.jpg temp/1748449887_2494884_1361521579_84ade84cfe497dac215b9a00d97d6c55_55084995.jpg temp/1748449887_2494884_1361521578_d82f8adb1490038eab00d32215e7ea1b_40652141.jpg temp/1748449887_2494884_1361521577_fc425337fd7327ae5e96e840680d5871_52238782.jpg temp/1748449887_2494884_1361521576_4b55835ded581086403cc33db67da82c_20085129.jpg temp/1748449887_2494884_1361521575_f42680a0d9782f2e75a1dfbec4b19761_71938533.jpg temp/1748449887_2494884_1361521573_bcc9c245ff85b10048d181456154ef0d_73489693.jpg temp/1748449887_2494884_1361521560_2be3194c0f8aeb15c90cc87029dc3af5_89898320.jpg temp/1748449887_2494884_1361521559_c3537bb9733bdcb0906bb78ab13a986c_5647720.jpg temp/1748449887_2494884_1361521558_675304bbb637957642fb150366b33037_21416230.jpg temp/1748449887_2494884_1361521557_e75e69ad5afe6b6aaf5a13981072880c_40652863.jpg temp/1748449887_2494884_1361521556_03b666cf03863c6ca332019de0b5e878_3850814.jpg temp/1748449887_2494884_1361521555_d9b974ae925ad2d2045744924541047e_60488729.jpg temp/1748449887_2494884_1361521553_79b5d59a8fcfc4f7f037f280eda5a9ef_30951914.jpg temp/1748449887_2494884_1361521552_8dc8550c2951b35bee8b7ab1e10731bf_61631826.jpg temp/1748449887_2494884_1361521551_c912ec4d6603dbcc1aa67e57ccf13775_57352910.jpg temp/1748449887_2494884_1361521550_c6c3cf92122c99b7f373c5d109c1dc5c_76138554.jpg temp/1748449887_2494884_1361521549_d86205e5ef644b0cb153be341aaa38ea_81988150.jpg temp/1748449887_2494884_1361521548_f9d445f6b3f528a5970d050d1378dbb2_165728.jpg temp/1748449887_2494884_1361521545_20cb239525d93432c9a0e26afe49d88a_51007557.jpg temp/1748449887_2494884_1361521544_63cd861091b60471bf28dd549c1acb92_47581186.jpg temp/1748449887_2494884_1361521543_68203aea79caaf821b11c52d239969b9_17781266.jpg temp/1748449887_2494884_1361521542_dcb0bf243969af1e6156a6f8a4dcebd4_38577496.jpg temp/1748449887_2494884_1361521541_8b3192994e478ed6587c50802e588f53_84439572.jpg temp/1748449887_2494884_1361521540_6bb3fa8a2816e8e2b683e27514735e14_88845563.jpg temp/1748449887_2494884_1361521537_76073d993d47a7ecd623ad0a4c7ef9a6_70492026.jpg temp/1748449887_2494884_1361521536_b61730ff71f5cae0fb5ced1461b42545_55896107.jpg temp/1748449887_2494884_1361521534_5430f76daba94bdb0e74c53234f428b2_57186741.jpg temp/1748449887_2494884_1361521532_94644cce8b4b5a3adc9dab66a4a0d571_1583798.jpg temp/1748449887_2494884_1361521530_6414789e36cdc0a8e879aaf242242440_14128198.jpg temp/1748449887_2494884_1361521524_27d4628fbd758bcd918628f92cf4d204_20284757.jpg temp/1748449887_2494884_1361521523_ad3f7c1f363e3abce82c3c92d5aa77b0_76600891.jpg temp/1748449887_2494884_1361521522_12ffac341aaf898ad11d816fd83fa59a_58039699.jpg temp/1748449887_2494884_1361521521_8e581836a6c3eb74b50e23b6ebc8c31d_34345949.jpg temp/1748449887_2494884_1361521520_5f1697b0ebdbde0dd1d7132400b0d1d4_77377112.jpg temp/1748449887_2494884_1361521512_f9530b13c47de5a43d56d6b291bf97d4_54454739.jpg temp/1748449887_2494884_1361521511_620b5dd81051c9f148b971f32ba7024c_75729189.jpg temp/1748449887_2494884_1361521510_a860a6f43795a9904c6db9a4b4cb8101_27507059.jpg temp/1748449887_2494884_1361521509_3457f38f6a3e0fd60fbc758f9eff5af2_70140546.jpg temp/1748449887_2494884_1361521508_3586e17fa73a24e0814dc86c72095f60_3524272.jpg temp/1748449887_2494884_1361521507_63b508516aa55fa58683f1f0fd4d614d_73229782.jpg temp/1748449887_2494884_1361521506_1e6c35bc81315a765977f2e809915c64_20677808.jpg temp/1748449887_2494884_1361521505_79f613bf846bbd8ef875ad1915227f97_2758447.jpg temp/1748449887_2494884_1361521504_099097905068bbd6a4a962430cea71f9_15058582.jpg temp/1748449887_2494884_1361521503_1d947a921edf0911ae57fa39751b9fd7_69739762.jpg temp/1748449887_2494884_1361521501_a1d2673ed7293bbb54cbfb5d083bd90b_43298156.jpg temp/1748449887_2494884_1361521499_f5e9ec855c2330f8c4fcd4cc159a891b_62661731.jpg temp/1748449887_2494884_1361521485_8669acbc1ed733874ee462e0d1bb75fe_77885495.jpg temp/1748449887_2494884_1361521484_36b93a84a0370583f4bd6da030fc655b_18935333.jpg temp/1748449887_2494884_1361521483_2eb6801857a6ed0e793b47ab3284e1af_49458263.jpg temp/1748449887_2494884_1361521482_45856b3af2a902029d2b7dbde5136a84_31019930.jpg temp/1748449887_2494884_1361521481_ccec639c070c80f20173f97902df3e65_25523251.jpg temp/1748449887_2494884_1361521480_124ecb1545d9df6445f011883d550558_74287392.jpg temp/1748449887_2494884_1361521479_a696fd53fb9b9f2cae9f4029da4eca14_20213960.jpg temp/1748449887_2494884_1361521477_cc6324b8c1019f4bee536a47f122b8f4_7454935.jpg temp/1748449887_2494884_1361521476_c010fbc7bf88c58fb5b6b6c1a8876e71_29573187.jpg temp/1748449887_2494884_1361521475_a08104e9d10a7346816672a17e58f0d3_97694466.jpg temp/1748449887_2494884_1361521474_8be0bfd8c03b8e2a1a74a68223a96a1c_76784399.jpg temp/1748449887_2494884_1361521473_9157608eb4583344daf1c5b24350d37e_94534227.jpg temp/1748449887_2494884_1361521455_2d8fe92691cf8901ac5979f5e81fdaaf_95647138.jpg temp/1748449887_2494884_1361521453_601056c0fda3587bfc3d9d5cc4de86c9_82788829.jpg temp/1748449887_2494884_1361521451_28ca573524e77229b72b828b036e6300_16893051.jpg temp/1748449887_2494884_1361521449_faf8d1382eab1de11575dc8de8435871_82632499.jpg temp/1748449887_2494884_1361521447_4ba01b1f86360ab7fd0a020fdd433f7c_29362777.jpg temp/1748449887_2494884_1361521445_c371c057a48abd3dce0e1494e3649ac2_58200435.jpg temp/1748449887_2494884_1361521442_559cdb039557cf5cfc8bab540e1899c4_97514897.jpg temp/1748449887_2494884_1361521441_2a001aedc50dce3f424242f6a43684f9_24751037.jpg temp/1748449887_2494884_1361521440_d98519e25e7689ba6ed142dda863f79a_21188971.jpg temp/1748449887_2494884_1361521439_efd271a7f6af9e2a31ffa46ff1a06c11_62545122.jpg temp/1748449887_2494884_1361521438_417c90960ef53c2e4a1530740f5f0248_31800671.jpg temp/1748449887_2494884_1361521437_32329da727ed19961886947b9dabc726_85944431.jpg temp/1748449887_2494884_1361521428_92630385b12ae86cb93b5c5e00913f95_77089083.jpg temp/1748449887_2494884_1361521427_fbf37f3f4cd7e7e646c7a14543b64543_33085938.jpg temp/1748449887_2494884_1361521426_e0b8ced3ef0b3a74919646103ada8e85_70159203.jpg temp/1748449887_2494884_1361521424_8b67b6f83225e0c756019f4b9c2b1cda_44680066.jpg temp/1748449887_2494884_1361521423_1890c9258a0a8fcf71ee05e4f018f06c_95173521.jpg temp/1748449887_2494884_1361521422_2fc82dc194be5724253b0c59de705dd8_51075265.jpg temp/1748449887_2494884_1361521421_f1920739caac41eb602ec62e9c972f4f_32262877.jpg temp/1748449887_2494884_1361521420_2b52065de0b27200f5cedb26adc9b99e_31859162.jpg temp/1748449887_2494884_1361521418_038abe338ac01216100e3988c34528ca_44401545.jpg temp/1748449887_2494884_1361521417_adbaeff71e24c3778b783ecddb98a6d7_90060550.jpg temp/1748449887_2494884_1361521416_2d96888b4434770f9c14f7a0bc1bccba_31242871.jpg temp/1748449887_2494884_1361521415_aee6978600ff525aa9a8c392c140e550_95101489.jpg temp/1748449887_2494884_1361521401_9f935dbf117b3746d62fe82e7d1200ea_60512774.jpg temp/1748449887_2494884_1361521399_2a434cf5185144c5992424298ad30eba_41654967.jpg temp/1748449887_2494884_1361521398_a06154d857c91ef9cbce036986da3a39_86414805.jpg temp/1748449887_2494884_1361521397_d2a76c833f51cf63fdd0d296e95761d0_84986892.jpg temp/1748449887_2494884_1361521396_909909f1837dde42afce3cbcf22a714c_21143134.jpg temp/1748449887_2494884_1361521395_ba452a2f8246cfd62b089529956f9133_4757157.jpg temp/1748449887_2494884_1361521384_b867dcd8ce702beef0f022354a48d030_80692683.jpg temp/1748449887_2494884_1361521383_55524986e5db61a09fe9af4ab1e31f03_3288213.jpg temp/1748449887_2494884_1361521382_3aa332031f4d76e47ad70f22e00ffb2c_47013133.jpg temp/1748449887_2494884_1361521381_3c87319b4b869fc7e761614f628f1c22_35895398.jpg temp/1748449887_2494884_1361521380_a73d1585436855b24b2ea56dcc79a1a0_37754610.jpg temp/1748449887_2494884_1361521379_8c625d2f4fb4b1930edab910b138ae0d_81006153.jpg temp/1748449887_2494884_1361521348_16cee56e6a778654e56da1f5ea1eabb5_82505975.jpg temp/1748449887_2494884_1361521345_2a640127a295ea07414d6bc10dbb1450_14033146.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361521587, 1361521585, 1361521583, 1361521581, 1361521579, 1361521578, 1361521577, 1361521576, 1361521575, 1361521573, 1361521560, 1361521559, 1361521558, 1361521557, 1361521556, 1361521555, 1361521553, 1361521552, 1361521551, 1361521550, 1361521549, 1361521548, 1361521545, 1361521544, 1361521543, 1361521542, 1361521541, 1361521540, 1361521537, 1361521536, 1361521534, 1361521532, 1361521530, 1361521524, 1361521523, 1361521522, 1361521521, 1361521520, 1361521512, 1361521511, 1361521510, 1361521509, 1361521508, 1361521507, 1361521506, 1361521505, 1361521504, 1361521503, 1361521501, 1361521499, 1361521485, 1361521484, 1361521483, 1361521482, 1361521481, 1361521480, 1361521479, 1361521477, 1361521476, 1361521475, 1361521474, 1361521473, 1361521455, 1361521453, 1361521451, 1361521449, 1361521447, 1361521445, 1361521442, 1361521441, 1361521440, 1361521439, 1361521438, 1361521437, 1361521428, 1361521427, 1361521426, 1361521424, 1361521423, 1361521422, 1361521421, 1361521420, 1361521418, 1361521417, 1361521416, 1361521415, 1361521401, 1361521399, 1361521398, 1361521397, 1361521396, 1361521395, 1361521384, 1361521383, 1361521382, 1361521381, 1361521380, 1361521379, 1361521348, 1361521345] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.051810503005981445 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 19.651182413101196 time spend to save output : 0.057175636291503906 total time spend for step 0 : 19.7083580493927 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.55user 4.99system 0:38.69elapsed 47%CPU (0avgtext+0avgdata 1942820maxresident)k 17440inputs+203432outputs (53major+657518minor)pagefaults 0swaps