python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3467206 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 19686>100 please execute split_portfolio.py -i 23282116 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2939103'] with mtr_portfolio_ids : ['23282116'] and first list_photo_ids : [] new path : /proc/3467206/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 86 ; length of list_pids : 86 ; length of list_args : 86 time to download the photos : 12.380744695663452 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri Sep 12 18:22:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-09-12 18:22:49.232649: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-12 18:22:49.264098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-12 18:22:49.264392: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-12 18:22:49.267514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-12 18:22:49.269494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-12 18:22:49.269796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-12 18:22:49.271907: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-12 18:22:49.272951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-12 18:22:49.277558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-12 18:22:49.278656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-12 18:22:49.279143: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-12 18:22:49.304572: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-12 18:22:49.306224: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7d4c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-12 18:22:49.306274: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-12 18:22:49.459605: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1e849d20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-12 18:22:49.459641: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-12 18:22:49.460437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-12 18:22:49.460537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-12 18:22:49.460557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-12 18:22:49.460574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-12 18:22:49.460590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-12 18:22:49.460607: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-12 18:22:49.460623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-12 18:22:49.460639: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-12 18:22:49.461439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-12 18:22:49.461483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-12 18:22:49.461993: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-12 18:22:49.462008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-12 18:22:49.462015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-12 18:22:49.462805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2940 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-09-12 18:22:56.354029: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-12 18:22:56.354668: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.066121578216553 time used to load_weights : 0.3209402561187744 0it [00:00, ?it/s] 86it [00:00, 1660.87it/s]2025-09-12 18:22:59.835739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-12 18:23:00.043112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-12 18:23:00.318374: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-09-12 18:23:00.320191: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1757694149_3467206_1383519430_ff0a6f41770ef6baa530c5f8669f989d.jpg temp/1757694149_3467206_1383519427_6ada26e2c7d9d702a1fb53a72408a880.jpg temp/1757694149_3467206_1383519420_e8d52772b1136c7d3b84154357c618e0.jpg temp/1757694149_3467206_1383519419_6d067493b68e781b75ea36f4402baf2a.jpg temp/1757694149_3467206_1383519418_87066e0ace235731cd6e2a2d679fe1f9.jpg temp/1757694149_3467206_1383519417_41ee5d635ce6c62e6b51329a1966c29a.jpg temp/1757694149_3467206_1383519294_783382022bfa560bf3a27d41f4dd338f.jpg temp/1757694149_3467206_1383519289_aecc4a6bb9f8e28d98d1a74958fcbd05.jpg temp/1757694149_3467206_1383519286_1fd824e69c61503d1e21207d51c47e8f.jpg temp/1757694149_3467206_1383519285_ac14a120fb05f08fd66e0f6c2c8df282.jpg temp/1757694149_3467206_1383519283_a4492bd03d6d49590ae81bbf4c02be45.jpg temp/1757694149_3467206_1383519277_0a710256eaa5a732fafe7c3695c05eb3.jpg temp/1757694149_3467206_1383519209_b8c54c0b92764cd85a8691226121b08a.jpg temp/1757694149_3467206_1383519203_20cea01afa53253ff7dc48235c5361b7.jpg temp/1757694149_3467206_1383519201_321de3119e4ea499fddb7d81398b6349.jpg temp/1757694149_3467206_1383519199_6c9843f16521e5723020dfa89eabad3d.jpg temp/1757694149_3467206_1383519197_cc1f54e22ec6e00b2213cc8f7187dc70.jpg temp/1757694149_3467206_1383519196_fd91dcdb61710ccb7875912fb84f6f9d.jpg temp/1757694149_3467206_1383519156_2ec02f907113c8bce3f948eabf3d71a3.jpg temp/1757694149_3467206_1383519152_86e947bfad80939258891cafa8620d9d.jpg temp/1757694149_3467206_1383519149_50fdc268d3480a95ecb4d647e690ce0b.jpg temp/1757694149_3467206_1383519146_d0617772adf58fa905cc289d1710efcc.jpg temp/1757694149_3467206_1383519143_7c2b5f534a44b5ca41e1989678ddf12d.jpg temp/1757694149_3467206_1383519140_d5181d3411446ec5d0f4e6ed90b7823f.jpg temp/1757694149_3467206_1383511562_b0973fdf2246f9a97965b6863e580388.jpg temp/1757694149_3467206_1383511560_06209258230ca418699059066e6ca684.jpg temp/1757694149_3467206_1383511374_8e3ee1c06e5e0417f2e7d687bdf8376f.jpg temp/1757694149_3467206_1383511347_2bf98c3ba1712c9dd83cb57d034b46b4.jpg temp/1757694149_3467206_1383511323_b56ba53f9fff94853de98cd265152359.jpg temp/1757694149_3467206_1383511299_a518813272f6f558ebc95e7a8f5cff79.jpg temp/1757694149_3467206_1383511274_14f335b4c1791b9551887708ba88dd1b.jpg temp/1757694149_3467206_1383511248_35785325c48c28ca9874e3a3313e65de.jpg temp/1757694149_3467206_1383511203_cc9da897e6c10a5f4502cad930546256.jpg temp/1757694149_3467206_1383511202_aa3eb567cd36de9f0d77f3f9790f3fbb.jpg temp/1757694149_3467206_1383511187_61c5ed82d8da20b5d134459c65a789b5.jpg temp/1757694149_3467206_1383511160_6e298d0840cf1438004ef9b76b7afa91.jpg temp/1757694149_3467206_1383511141_04a72c817a8c190a00777ae1da874952.jpg temp/1757694149_3467206_1383511121_bb6bd93b4c9c5e2fa6908e632542d04c.jpg temp/1757694149_3467206_1383511070_ff357b7b4c6af16760db433ce88992d7.jpg temp/1757694149_3467206_1383511043_1f9319eb616862f5df719f4026f61b75.jpg temp/1757694149_3467206_1383511019_44694a93b08538ca47b53cd44d0fa591.jpg temp/1757694149_3467206_1383510989_40274472f50f0046086e813b5125ffca.jpg temp/1757694149_3467206_1383510966_2afa6ce466aeb04c388abf5b4d0d5472.jpg temp/1757694149_3467206_1383510921_eb345e8eec82056c1bb6bb5c646d5adb.jpg temp/1757694149_3467206_1383510632_82c9bb17735cc78d0092a30cfc95e74b.jpg temp/1757694149_3467206_1383510606_2c256da9fb04e7806ccdc84317168681.jpg temp/1757694149_3467206_1383510597_200f3e78183ec8c1e1d1b86725e8ed81.jpg temp/1757694149_3467206_1383510595_25d71b07607ad447fdee2c74acc723b6.jpg temp/1757694149_3467206_1383510590_24a3362871c14b7bfb2411576e197a8b.jpg temp/1757694149_3467206_1383510571_ddc71998a3b955b2e87ca0fafc9f3ae8.jpg temp/1757694149_3467206_1383510305_24d022cf7a944367114b3fe5f4a4a170.jpg temp/1757694149_3467206_1383510279_592e10fea5e36242b17b9941c5389c6f.jpg temp/1757694149_3467206_1383510251_c6fec902ffca6668623388e50ebb4454.jpg temp/1757694149_3467206_1383510226_0cbc3af6419899dd638f678aece8b279.jpg temp/1757694149_3467206_1383510200_7f83d20135c0cc678c0ef25897c1fecd.jpg temp/1757694149_3467206_1383510174_0a06c7732cae01fc1bd2ba28150fa0db.jpg temp/1757694149_3467206_1383509282_2ff84722fd2889738799f5f5cfa32ee2.jpg temp/1757694149_3467206_1383509278_7b9896626481dc3fc9c7f50cc8a6b47f.jpg temp/1757694149_3467206_1383509275_65746189ad520354f7f270748fcb0ecc.jpg temp/1757694149_3467206_1383509272_b1419fd35cada5ed04f21c78ea47894b.jpg temp/1757694149_3467206_1383509263_2e5561ab3c4368281c8cea9d2845d9c2.jpg temp/1757694149_3467206_1383509247_75acba4355e4e448b50590bfd6a3ecac.jpg temp/1757694149_3467206_1383509007_926e8996cc1326a2e8bb30594e6c450e.jpg temp/1757694149_3467206_1383508995_c840bfc44ac0b0e58d99eea318e531e8.jpg temp/1757694149_3467206_1383508993_dd6ac37086d524cd03537b3e57502823.jpg temp/1757694149_3467206_1383508992_a3e36c259876c4e07d29b8dfd90f54b9.jpg temp/1757694149_3467206_1383508991_24900b5afae8fda60c095cc1fcda4bd5.jpg temp/1757694149_3467206_1383508989_c466f1ebe473edab666e93fb2175b5f7.jpg temp/1757694149_3467206_1383508777_e61c73eca16964e0719f013de5d47f1d.jpg temp/1757694149_3467206_1383508726_9f5ad90b523c553f95298336ac436d84.jpg temp/1757694149_3467206_1383508665_5f82ce4fdcfed166e90483f4698b0ad2.jpg temp/1757694149_3467206_1383508609_24c6032f6bacdf110bcc0637a0463a7e.jpg temp/1757694149_3467206_1383508559_39f7f1b225bb54183cc1d4058f2f5fe5.jpg temp/1757694149_3467206_1383508535_b152d7350b9e0200223b8ec24af7d7c6.jpg temp/1757694149_3467206_1383508352_c8c45e857b13ef8d8aa767f9665e680c.jpg temp/1757694149_3467206_1383508348_fe5e5942735d4706565e7eff34814016.jpg temp/1757694149_3467206_1383508344_a7e7acb8f504e2059f4d4d678f987aab.jpg temp/1757694149_3467206_1383508339_1b190d85b0b0f4137aecd0bc60532fd6.jpg temp/1757694149_3467206_1383508319_30b7a23ac074a6f97a472319eb9b36f6.jpg temp/1757694149_3467206_1383508296_d46aadfb4a935dad87e12efe2943c35d.jpg temp/1757694149_3467206_1383508131_63cd470005aeb1dc85b79bb17fd8737f.jpg temp/1757694149_3467206_1383508129_98a71a2c7d63e70699a16b157cffa1f7.jpg temp/1757694149_3467206_1383508127_7598f1c33f83df7d563a4491ab231954.jpg temp/1757694149_3467206_1383508126_e216b4ded3e069a31b39e1376cc55596.jpg temp/1757694149_3467206_1383508125_74bf49a634db9e74e5bd8f14a60d0196.jpg temp/1757694149_3467206_1383508112_72ef5ef4c5536c1161152e078f722d17.jpg Found 86 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1383519430, 1383519427, 1383519420, 1383519419, 1383519418, 1383519417, 1383519294, 1383519289, 1383519286, 1383519285, 1383519283, 1383519277, 1383519209, 1383519203, 1383519201, 1383519199, 1383519197, 1383519196, 1383519156, 1383519152, 1383519149, 1383519146, 1383519143, 1383519140, 1383511562, 1383511560, 1383511374, 1383511347, 1383511323, 1383511299, 1383511274, 1383511248, 1383511203, 1383511202, 1383511187, 1383511160, 1383511141, 1383511121, 1383511070, 1383511043, 1383511019, 1383510989, 1383510966, 1383510921, 1383510632, 1383510606, 1383510597, 1383510595, 1383510590, 1383510571, 1383510305, 1383510279, 1383510251, 1383510226, 1383510200, 1383510174, 1383509282, 1383509278, 1383509275, 1383509272, 1383509263, 1383509247, 1383509007, 1383508995, 1383508993, 1383508992, 1383508991, 1383508989, 1383508777, 1383508726, 1383508665, 1383508609, 1383508559, 1383508535, 1383508352, 1383508348, 1383508344, 1383508339, 1383508319, 1383508296, 1383508131, 1383508129, 1383508127, 1383508126, 1383508125, 1383508112] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 86 time used for this insertion : 0.053986310958862305 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 19.00932288169861 time spend to save output : 0.060454607009887695 total time spend for step 0 : 19.069777488708496 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.16user 4.06system 0:36.43elapsed 47%CPU (0avgtext+0avgdata 2041032maxresident)k 232992inputs+140640outputs (216major+606784minor)pagefaults 0swaps