python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2556624 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 8520>100 please execute split_portfolio.py -i 22866006 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2818761'] with mtr_portfolio_ids : ['22866006'] and first list_photo_ids : [] new path : /proc/2556624/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 14.30191683769226 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Tue Sep 2 10:34:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-09-02 10:34:50.522292: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-02 10:34:50.555494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-02 10:34:50.556762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-02 10:34:50.585433: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-02 10:34:50.592850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-02 10:34:50.597423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-02 10:34:50.601190: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-02 10:34:50.602328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-02 10:34:50.608520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-02 10:34:50.609640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-02 10:34:50.610154: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-02 10:34:50.636728: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-02 10:34:50.638651: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe300000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-02 10:34:50.638717: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-02 10:34:50.775262: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xc510ff0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-02 10:34:50.775327: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-02 10:34:50.776630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-02 10:34:50.776746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-02 10:34:50.776775: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-02 10:34:50.776801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-02 10:34:50.776827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-02 10:34:50.776852: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-02 10:34:50.776877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-02 10:34:50.776904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-02 10:34:50.777890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-02 10:34:50.777949: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-02 10:34:50.778610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-02 10:34:50.778632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-02 10:34:50.778645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-02 10:34:50.779703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3039 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-09-02 10:34:57.825224: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.273213386535645 time used to load_weights : 0.30820584297180176 0it [00:00, ?it/s] 100it [00:00, 1477.17it/s]2025-09-02 10:35:01.547135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-02 10:35:02.015537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-02 10:35:02.787846: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-09-02 10:35:02.811619: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1756802067_2556624_1380805329_068553e3dabb18b496442d0591f4b471_12678441.jpg temp/1756802067_2556624_1380805327_f19214d6801e2acc14d9a33492f85410_36592451.jpg temp/1756802067_2556624_1380805325_a59489ca956289db9b0da658e75db5f6_95454718.jpg temp/1756802067_2556624_1380805322_98dc792f5835ebdfaed781128b98afa7_83377707.jpg temp/1756802067_2556624_1380805320_631f739ae834537dffd254e7b9814b1d_19519263.jpg temp/1756802067_2556624_1380805318_aea12d8cfd56fe0e344de39208c39423_66561121.jpg temp/1756802067_2556624_1380805264_6b1942847e0bc110ed1beba32c59458a_12318186.jpg temp/1756802067_2556624_1380805263_96b75b64d8bd4c10df5def16ebdc5e84_81976790.jpg temp/1756802067_2556624_1380805262_82ed70cea7ab9b79f4e96aeb9fbb7089_69242622.jpg temp/1756802067_2556624_1380805261_e893bfb87b7e1be93b63a43747643b75_57443038.jpg temp/1756802067_2556624_1380805260_880f368813bc4dcc79adad794b58361b_4380990.jpg temp/1756802067_2556624_1380805259_6021c68cb37558aa4e72ea586e9cc80a_44459601.jpg temp/1756802067_2556624_1380805203_e78cd03fb5d4a1d3e4dff7fcc55770ad_60795445.jpg temp/1756802067_2556624_1380805202_fca886d4a11aceead063545996ec8ebb_26538150.jpg temp/1756802067_2556624_1380805201_0d9e3562c7e3cd9951593512823bd618_85872545.jpg temp/1756802067_2556624_1380805200_2cf35de1e6e349335fff3ff4258c0895_39484091.jpg temp/1756802067_2556624_1380805199_3e2f56e87ff1a4f88abc97d6e95b4abc_91533727.jpg temp/1756802067_2556624_1380805198_70a6fb3afc6c1fb988d5d3714a49a535_51372555.jpg temp/1756802067_2556624_1380805182_63212bf0142e2722fd28dd9174001d0a_64380855.jpg temp/1756802067_2556624_1380805180_164079c112c7fdfe0ef0f92a4a64484e_3064661.jpg temp/1756802067_2556624_1380805179_6adca5a088feeda6e9d9d4f53fe279cf_49408576.jpg temp/1756802067_2556624_1380805178_c56aed31c6d2b99aaa30620980c528e3_75445193.jpg temp/1756802067_2556624_1380805177_62044b0f6dc3ada8d6fef60992ce871e_16422163.jpg temp/1756802067_2556624_1380805176_e979637432ee88acff141324a98fb7da_11872476.jpg temp/1756802067_2556624_1380805164_ed54ca514e3e9799af90e836ec81bd60_64637520.jpg temp/1756802067_2556624_1380805163_acbc308012cf820e089f11d5016cc4d7_16829068.jpg temp/1756802067_2556624_1380805162_14c38f411beeb8d1fb36fbcd91f104c4_3965523.jpg temp/1756802067_2556624_1380805161_5b152b68480154199d447a6a15261984_63212579.jpg temp/1756802067_2556624_1380805160_8962194ee8c2a10b181990445dd7be83_13529292.jpg temp/1756802067_2556624_1380805159_9422519688df53d7178f0eb1409dfb1f_78828841.jpg temp/1756802067_2556624_1380805140_b65202b663a387cf3e81694d8e0b98bf_62510339.jpg temp/1756802067_2556624_1380805139_0a661591953409ba8186504ec32a7b90_60199068.jpg temp/1756802067_2556624_1380805138_8577ca1debef282358e794bad2b5de18_30763015.jpg temp/1756802067_2556624_1380805137_b8559ab3b44ae780960de3bc0c62d116_38042734.jpg temp/1756802067_2556624_1380805136_e880bfc301d08b65c1232337ef9d3bc4_65067696.jpg temp/1756802067_2556624_1380805135_ba7a10b10d6623341f0a227dc74a34fa_48287190.jpg temp/1756802067_2556624_1380805105_ce1f3e310f4479ed18266d4ba2074eb1_4651655.jpg temp/1756802067_2556624_1380805104_9e072c25a392e1f97f4091d2cdf6f538_6629144.jpg temp/1756802067_2556624_1380805103_bd238284d9256dc4b19f2526f07a8753_96241491.jpg temp/1756802067_2556624_1380805102_27b89db6a2a007e26ef177f3ca460c3f_28508203.jpg temp/1756802067_2556624_1380805101_ba80ca3208ae259782c354c199a3ece4_46160340.jpg temp/1756802067_2556624_1380805100_6928a4863c3c832e2848525891330fe5_55199447.jpg temp/1756802067_2556624_1380804997_25182f14f39b56fffa0d336280b4e17c_35985484.jpg temp/1756802067_2556624_1380804995_eccdbc1de89a2c1b0a1e22825b0c48fa_20030147.jpg temp/1756802067_2556624_1380804994_7fd58fec8904f18a38a5b2fb08da10c1_39313576.jpg temp/1756802067_2556624_1380804993_374c36529de2ce29518845455499039d_44950711.jpg temp/1756802067_2556624_1380804992_90e76ea28dc258977cbe44e3ef7d5d51_25697040.jpg temp/1756802067_2556624_1380804991_4d6bba199b648ad56618c7695b200c85_67257038.jpg temp/1756802067_2556624_1380804955_8dbee179faaf166bb48868cb82a1076a_41157504.jpg temp/1756802067_2556624_1380804953_29c99da0f9d86f236eb40bbe9e5b68f3_63328421.jpg temp/1756802067_2556624_1380804951_a0ad58710833416ca0d2bc01c4b1e9df_27407482.jpg temp/1756802067_2556624_1380804949_bf26f756d506bd11267e9b2b5c912b3e_42837215.jpg temp/1756802067_2556624_1380804947_a63fcdf5f1c179dfc81735c6073c9aa5_4178461.jpg temp/1756802067_2556624_1380804945_a99be47f2943cc0e60a2e98456b73c95_91489495.jpg temp/1756802067_2556624_1380804808_12170f956c00365f05ebe68e4333d176_17258823.jpg temp/1756802067_2556624_1380804806_875bdd9f3d1504948140ce8a650b7bfe_65695381.jpg temp/1756802067_2556624_1380804804_dafec13f7597d9aa92a593c087ccddb9_70272043.jpg temp/1756802067_2556624_1380804802_d20e6adb033b2f65aeba8597d9101a66_54710678.jpg temp/1756802067_2556624_1380804801_ccdaf8248e486068117df90d2e76f411_78963244.jpg temp/1756802067_2556624_1380804799_24255c83d9493dca4249abe8ff518e33_89801454.jpg temp/1756802067_2556624_1380804681_42e712099ce02100fe44e752f359e334_79287292.jpg temp/1756802067_2556624_1380804679_1821933bed3cef1426ec351ba5f4e018_51327374.jpg temp/1756802067_2556624_1380804677_85c79ea9366a4c1e138ca1f4a7b9960d_46715588.jpg temp/1756802067_2556624_1380804675_7029a865b2d278109cfee0da81a61b67_21755024.jpg temp/1756802067_2556624_1380804673_419f32d2e9d4e6f3384b675b07b8b314_90463639.jpg temp/1756802067_2556624_1380804670_096781135ce402d059ff5bdf1f87773e_57878990.jpg temp/1756802067_2556624_1380804564_65b0a4c44cb73a76d027784ce14234e7_9205675.jpg temp/1756802067_2556624_1380804563_c92183ef747ccd433df31b2f09b34355_6821041.jpg temp/1756802067_2556624_1380804562_a01c5e06eba9e6a8ceabd024d2de686c_57341832.jpg temp/1756802067_2556624_1380804561_151f7c729e1428fb79a6b34866d5e118_99015898.jpg temp/1756802067_2556624_1380804560_f7c1dcde6747f640032b918704ee0d9c_17529446.jpg temp/1756802067_2556624_1380804559_658ce115de5eab0280b04744d8108d2f_34793732.jpg temp/1756802067_2556624_1380804501_3bce6f732433f39a1d86f4faf05f0e53_11699668.jpg temp/1756802067_2556624_1380804500_ed2e7ebeb80ec1c477f3799c59cca777_68088572.jpg temp/1756802067_2556624_1380804499_011af77489fcc921433f83f6bab1d3d4_59837866.jpg temp/1756802067_2556624_1380804498_4f9bded9d5833c7e6c0ca3cb65029342_60231916.jpg temp/1756802067_2556624_1380804497_ed24cd013ef6352a84d314627f83c59a_30671061.jpg temp/1756802067_2556624_1380804496_8579b14ba25b19717421424fb48be212_2397662.jpg temp/1756802067_2556624_1380804445_457cf3abf17ca2626cc820b6ae4d5240_89012718.jpg temp/1756802067_2556624_1380804444_06bfde87a223fef7da97776f97f23b38_20869362.jpg temp/1756802067_2556624_1380804443_b18376c687fa6880d3b1a0b981256585_65499517.jpg temp/1756802067_2556624_1380804442_23ddaa4fb97d5540d3ca058e8218b540_89486544.jpg temp/1756802067_2556624_1380804441_7e2d6d0a2b73a9ee51a997ab5212680b_48733640.jpg temp/1756802067_2556624_1380804440_3a89c14ed137a010adb16cb83128bb64_51334146.jpg temp/1756802067_2556624_1380804414_25853cb80932ee859f684b9a1a385e36_49951975.jpg temp/1756802067_2556624_1380804413_a0887b8a12988edd388249274e45366e_37180097.jpg temp/1756802067_2556624_1380804412_f757476248f6e2c5a9e7ed8a27069c3a_79446168.jpg temp/1756802067_2556624_1380804411_ca0b614082cf923e8c917514052e6dfb_8069396.jpg temp/1756802067_2556624_1380804410_9a80602c7c1cc4dcc7a413a69018c099_19813083.jpg temp/1756802067_2556624_1380804409_fd67452828b8555a68952e46ffb037f0_18792188.jpg temp/1756802067_2556624_1380804321_a75a956c908f5df1071f07f2df6bc173_1201935.jpg temp/1756802067_2556624_1380804319_6e195f8d26ad865f2b816a09a40f45ec_15519104.jpg temp/1756802067_2556624_1380804317_7340e45e784dffb0e59f757d0e3a4851_37921058.jpg temp/1756802067_2556624_1380804315_f9816496d46489a30bdb6deb350d56da_22127624.jpg temp/1756802067_2556624_1380804312_86bb69a9e8b9b5266e3fe198e23a6b65_15830814.jpg temp/1756802067_2556624_1380804310_6f8dfff015d422a340b0f305d60ca288_18185783.jpg temp/1756802067_2556624_1380804143_e79d4d12536d7cb2218f7bfbac95cb45_35201806.jpg temp/1756802067_2556624_1380804139_71184f7e5adb9d1a8532f4a4ea23d9bb_84281915.jpg temp/1756802067_2556624_1380804136_be7f3f822996d8f03223474efe2059d2_902565.jpg temp/1756802067_2556624_1380804132_ba4988f0190fee9d0f89e431bda6ac74_19264242.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1380805329, 1380805327, 1380805325, 1380805322, 1380805320, 1380805318, 1380805264, 1380805263, 1380805262, 1380805261, 1380805260, 1380805259, 1380805203, 1380805202, 1380805201, 1380805200, 1380805199, 1380805198, 1380805182, 1380805180, 1380805179, 1380805178, 1380805177, 1380805176, 1380805164, 1380805163, 1380805162, 1380805161, 1380805160, 1380805159, 1380805140, 1380805139, 1380805138, 1380805137, 1380805136, 1380805135, 1380805105, 1380805104, 1380805103, 1380805102, 1380805101, 1380805100, 1380804997, 1380804995, 1380804994, 1380804993, 1380804992, 1380804991, 1380804955, 1380804953, 1380804951, 1380804949, 1380804947, 1380804945, 1380804808, 1380804806, 1380804804, 1380804802, 1380804801, 1380804799, 1380804681, 1380804679, 1380804677, 1380804675, 1380804673, 1380804670, 1380804564, 1380804563, 1380804562, 1380804561, 1380804560, 1380804559, 1380804501, 1380804500, 1380804499, 1380804498, 1380804497, 1380804496, 1380804445, 1380804444, 1380804443, 1380804442, 1380804441, 1380804440, 1380804414, 1380804413, 1380804412, 1380804411, 1380804410, 1380804409, 1380804321, 1380804319, 1380804317, 1380804315, 1380804312, 1380804310, 1380804143, 1380804139, 1380804136, 1380804132] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.07729840278625488 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 21.030938863754272 time spend to save output : 0.08385443687438965 total time spend for step 0 : 21.114793300628662 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.42user 5.30system 0:39.89elapsed 49%CPU (0avgtext+0avgdata 2429644maxresident)k 693888inputs+190320outputs (1825major+664884minor)pagefaults 0swaps