python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2640400 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 18092>200 please execute split_portfolio.py -i 22986467 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2855638'] with mtr_portfolio_ids : ['22986467'] and first list_photo_ids : [] new path : /proc/2640400/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 33.43800354003906 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri Sep 5 01:41:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-09-05 01:41:07.604825: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-05 01:41:07.637256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-05 01:41:07.637489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-05 01:41:07.639282: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-05 01:41:07.641034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-05 01:41:07.641328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-05 01:41:07.643334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-05 01:41:07.644361: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-05 01:41:07.648422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-05 01:41:07.649688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-05 01:41:07.650064: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-05 01:41:07.676559: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-05 01:41:07.678271: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f3fcc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-05 01:41:07.678299: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-05 01:41:07.787698: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11049e00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-05 01:41:07.787748: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-05 01:41:07.788844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-05 01:41:07.788948: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-05 01:41:07.788981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-05 01:41:07.789015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-05 01:41:07.789045: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-05 01:41:07.789075: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-05 01:41:07.789105: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-05 01:41:07.789135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-05 01:41:07.790878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-05 01:41:07.790948: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-05 01:41:07.791970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-05 01:41:07.791993: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-05 01:41:07.792007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-05 01:41:07.793877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 6819 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.993983030319214 time used to load_weights : 0.16434383392333984 0it [00:00, ?it/s]temp/1757029231_2640400_1381540573_eb9ff2c5f68d6ad520dc0bbca2dd7644_3193689.jpg temp/1757029231_2640400_1381540569_1c2ef5e2d285d68b8521270c585e081e_30021589.jpg temp/1757029231_2640400_1381540565_3d1506f1111bfdee169374c5678cb559_81802469.jpg temp/1757029231_2640400_1381540561_63e1ef7e75c203cb2572fde2df2872a2_94219668.jpg temp/1757029231_2640400_1381540557_2513d4d816e7d03b88edc1c87c138edf_55345665.jpg temp/1757029231_2640400_1381540553_e71b4d8485c5567c255c45099e3e7e99_31042990.jpg temp/1757029231_2640400_1381540328_eeb6edc2ffad1c28de674ef79f24a956_65914427.jpg temp/1757029231_2640400_1381540324_1b9b0bb70a85fd5ecf7faaf7bcfcf2dd_35266864.jpg temp/1757029231_2640400_1381540320_e2cefd3b6e894bf3e1b104c96205ec22_53652018.jpg temp/1757029231_2640400_1381540316_8cd767d08d2bdde50deb13dd728ac1bb_49914504.jpg temp/1757029231_2640400_1381540312_a6eeeca1dcd6edcd6471f92554879874_77468551.jpg temp/1757029231_2640400_1381540308_6e808482c703174b57a05ff57b5460cf_91868837.jpg temp/1757029231_2640400_1381540118_537f3b749c4e95a87371aa8f9cd6d516_16104017.jpg temp/1757029231_2640400_1381540115_1b0ef6a1a6017aed4c2f768e02a2b4e9_63695220.jpg temp/1757029231_2640400_1381540112_48774dd7c3e6a8f1790373946e4f3ef7_54666721.jpg temp/1757029231_2640400_1381540109_1b69ce9f16ed3e6f56019be9a71dee0b_37505613.jpg temp/1757029231_2640400_1381540106_44ff48c7e026fd223ea38cc10c4b2800_25257707.jpg temp/1757029231_2640400_1381540103_1264aaf3a8c3b0fd960fdb61534db8fb_86179664.jpg temp/1757029231_2640400_1381539716_75f9c27be1f7adad630f92602ffa2b69_69656233.jpg temp/1757029231_2640400_1381539712_dff8d775605aa61161e0ea1984023628_69492110.jpg temp/1757029231_2640400_1381539708_259163752f859160729d7a99857b3d28_76672140.jpg temp/1757029231_2640400_1381539704_e7272e0aaf2cf5649080cf711591e51d_90350712.jpg temp/1757029231_2640400_1381539700_8be2ec51139ae54b1ad1aa57540320cb_49578943.jpg temp/1757029231_2640400_1381539696_a85ba66bc7ed285cd6151399d65beeb5_62995243.jpg temp/1757029231_2640400_1381539355_d606987e8e01d72c9ecf7f7cfa568bbe_41923410.jpg temp/1757029231_2640400_1381539350_485c4d8c0151e8ab5ff1c2e6a2a1a6e8_4345407.jpg temp/1757029231_2640400_1381539344_0702b3fa57dab8aed2a6d8a97f43e297_13883784.jpg temp/1757029231_2640400_1381539339_848d8477821f4099ded136ffc85c9e4c_84918540.jpg temp/1757029231_2640400_1381539334_e39c9e851bf02ec4ccbb137e4ea73ce7_65169091.jpg temp/1757029231_2640400_1381539329_101a03b1b21de09c13ec763dcf2e20d0_714209.jpg temp/1757029231_2640400_1381539034_1236a3a3e87595a6cab0944acb24034a_3888793.jpg temp/1757029231_2640400_1381539030_eee3301b533bf64553a94c65e3223ad0_16949499.jpg temp/1757029231_2640400_1381539023_9c0673036f0ee3a40b270f759db9eda2_64380634.jpg temp/1757029231_2640400_1381539018_fdeb0d490d0eca2c69d720f395982b9d_88108853.jpg temp/1757029231_2640400_1381539013_a21bf58fb84a2dac5905155f310d2005_96035151.jpg temp/1757029231_2640400_1381539008_9618a458c397a08436548b536aef02ca_11199011.jpg temp/1757029231_2640400_1381538775_14a52977a071764726d56b280348b996_846149.jpg temp/1757029231_2640400_1381538770_1a38c04fbc58d71f0d0d697517ec99bf_63542683.jpg temp/1757029231_2640400_1381538765_8aa3d6412ff6270cd6b95fe893f7da11_64660520.jpg temp/1757029231_2640400_1381538760_5bc3ef437f9af113abb3d256fe9037a6_1179644.jpg temp/1757029231_2640400_1381538755_706013d7385c744424c2ab2fee64b06c_71049512.jpg temp/1757029231_2640400_1381538750_f0f120ab55fe445ddb6eef794190edd6_70939234.jpg temp/1757029231_2640400_1381538462_44b65cbdceea293058a516c102de315e_38672228.jpg temp/1757029231_2640400_1381538458_9737d12e32d9673df7abfaedbb855c37_60549627.jpg temp/1757029231_2640400_1381538454_460a8619fb5ababbc711c9fd105abf45_62423605.jpg temp/1757029231_2640400_1381538450_19da81dae404a2b9b5608c762cdc7f7f_39259426.jpg temp/1757029231_2640400_1381538446_68f0ee6ee58861fcffa33c7d06202b01_74304078.jpg temp/1757029231_2640400_1381538442_4938fa70013b0ad13874fd8f4ddfbbc4_5581133.jpg temp/1757029231_2640400_1381538046_162627d34cc23b1375741eef6f639b2f_30731983.jpg temp/1757029231_2640400_1381538041_12750900b7b7df128aae68d92c263627_22704771.jpg temp/1757029231_2640400_1381538037_ac756b285e4ff6fa52a17c42885bdffa_45451617.jpg temp/1757029231_2640400_1381538034_acf6a696baf679e3439167799a798140_31234452.jpg temp/1757029231_2640400_1381538031_33cabc0865a5ba45d77e65bc8c49ad09_93192513.jpg temp/1757029231_2640400_1381538027_ae4adffdca55c6c7ada2cc8aad4ba60f_41783004.jpg temp/1757029231_2640400_1381537921_2bb84fd89507099fcbd08a6450626ba4_17008082.jpg temp/1757029231_2640400_1381537918_97df8872bbbeb01eb3d744fae8368cb9_53175505.jpg temp/1757029231_2640400_1381537915_46a103590fd0f311fcc2b59615ecf7da_19803319.jpg temp/1757029231_2640400_1381537913_3f7f1a35b7e1e597e64812d74fd96830_60024561.jpg temp/1757029231_2640400_1381537911_bb9fe59872eda1f9c939fb3482bbd87e_29911589.jpg temp/1757029231_2640400_1381537909_50e2caf1aca8d48212e1add89a11d721_83576276.jpg temp/1757029231_2640400_1381537804_9d3d7a5fccdc9e25b51454bd39752442_4187884.jpg temp/1757029231_2640400_1381537802_7a121ea775229ea671bab5e2fe8282af_20734042.jpg temp/1757029231_2640400_1381537800_0c6dc5256dc2b57a04e5dc9548196977_54570016.jpg temp/1757029231_2640400_1381537798_5752490768caa78935acc191657fe0e7_44573596.jpg temp/1757029231_2640400_1381537796_1b9ce44fc6aaab760a629aa45cd02df2_84111218.jpg temp/1757029231_2640400_1381537794_2d69fd8bfd05e337b48e6e65d564c944_34379149.jpg temp/1757029231_2640400_1381537711_072e8a6fa4c5a1fa3381fcfb991ccb09_59280664.jpg temp/1757029231_2640400_1381537709_2ebd113806852ec194ff40d517be4415_46270129.jpg temp/1757029231_2640400_1381537707_86863b776d553c15f6c83c264e5276c1_98371968.jpg temp/1757029231_2640400_1381537705_3a2d6dd6c3b210f20c0df47119188dcf_1166498.jpg temp/1757029231_2640400_1381537702_afa60f054bf3214e80e2a0eabb63a7e1_84594228.jpg temp/1757029231_2640400_1381537699_df3065fb1988ba8f037220cc8fcd416e_24423058.jpg temp/1757029231_2640400_1381537610_bab3e26c943207c27c1c48a4cce5c43e_92232310.jpg temp/1757029231_2640400_1381537607_803222d29369132e11ccef4373e10d7f_7883824.jpg temp/1757029231_2640400_1381537604_0fc1fb59ce26aaa6cef3bef22a1b4c96_85171411.jpg temp/1757029231_2640400_1381537601_fda405ea73296f2639967a1ef3abc59e_20924245.jpg temp/1757029231_2640400_1381537598_318d80547f7ee5e3ecb54b3b9d485a77_87512307.jpg temp/1757029231_2640400_1381537595_df98db3bf6479771ccd227e2fb7a5259_19126438.jpg temp/1757029231_2640400_1381537550_2e8bab597a196f313734c408f9a8835e_54383951.jpg temp/1757029231_2640400_1381537549_ac24ac44972b02613b2c2ac05d15dc4c_20173006.jpg temp/1757029231_2640400_1381537548_8b0a06d46391c288025d6d95c6fdd289_34241173.jpg temp/1757029231_2640400_1381537547_f3c7a3ff456fd37474122c340ff47cfa_70366824.jpg temp/1757029231_2640400_1381537546_b9103c7558448e50a564228be83cc800_99268997.jpg temp/1757029231_2640400_1381537545_790618559970d7523fcfa17e5030d624_66045891.jpg temp/1757029231_2640400_1381537540_66a2875745dbebea2f0cfc5f030b5db0_42200227.jpg temp/1757029231_2640400_1381537539_e220ed6c263c403cac7ad3530e26a46f_7568026.jpg temp/1757029231_2640400_1381537538_8d4d6c31474d06d8b9eb94d3a97ef55e_66788814.jpg temp/1757029231_2640400_1381537537_60b8e0ae08b1eaeba011657bf81c221f_53208509.jpg temp/1757029231_2640400_1381537536_7ea88e6380b55dd757eeaa7f2ae1d882_18063112.jpg temp/1757029231_2640400_1381537535_cb95b8568b34678650bf7778894cf6de_86042859.jpg temp/1757029231_2640400_1381537507_83c884c42222cb948f12616e3b3a0f07_60715992.jpg temp/1757029231_2640400_1381537506_903b31bc29a6a2e0d5e8ee4a69afb78b_66114115.jpg temp/1757029231_2640400_1381537505_3568d49973c8dada83f37466e0f54087_32257818.jpg temp/1757029231_2640400_1381537504_100424bacb1afab684ddb1d73004c676_9270370.jpg temp/1757029231_2640400_1381537503_57df952d75b32ac3135afdf62c738f74_11433094.jpg temp/1757029231_2640400_1381537502_2a519ff85cabbb292c98fc6af21fdf97_3632450.jpg temp/1757029231_2640400_1381537447_4665d84a516994c374f20db5bcef72ca_73538592.jpg temp/1757029231_2640400_1381537446_6a6e32ea6cded03dd85af5fde6790016_40329374.jpg temp/1757029231_2640400_1381537445_56df00b440029b1b254351ed82b8921d_77374494.jpg temp/1757029231_2640400_1381537444_6b0b28448f668b9fcca83794ba29afe9_69457578.jpg temp/1757029231_2640400_1381537443_a02cc33b270723a0b2292884e3b12603_80563065.jpg 159it [00:00, 1583.70it/s] 200it [00:00, 1620.44it/s]2025-09-05 01:41:18.022252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-05 01:41:18.214104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-05 01:41:19.795798: E tensorflow/stream_executor/cuda/cuda_driver.cc:910] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-09-05 01:41:19.795841: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-09-05 01:41:19.795851: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-09-05 01:41:19.795866: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-09-05 01:41:19.795877: E tensorflow/stream_executor/stream.cc:5485] Internal: Failed to enqueue async memset operation: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-09-05 01:41:19.795888: W tensorflow/core/kernels/gpu_utils.cc:69] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-09-05 01:41:19.795897: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-09-05 01:41:19.795907: I tensorflow/stream_executor/stream.cc:4963] [stream=0x11ebea90,impl=0x11ebdb60] did not memzero GPU location; source: 0x7f3edf7fc020 temp/1757029231_2640400_1381537442_ce7e358685817c94834738d27b30b6c5_84121806.jpg temp/1757029231_2640400_1381537370_af6eed392c72db23c86c7d750f5c5f96_87839466.jpg temp/1757029231_2640400_1381537369_f8909cb97dd1ec5e8637f75ce1e023d7_17169319.jpg temp/1757029231_2640400_1381537368_3151558ed11deeb1c793d8f03d3f5a84_91827164.jpg temp/1757029231_2640400_1381537367_b44e310abb87baeee9b8d9a80e8097df_1327459.jpg temp/1757029231_2640400_1381537366_951dc5ba54f269de3bfd0660a6920036_11901251.jpg temp/1757029231_2640400_1381537365_7e617696031f937339dc5e34d3115540_27811964.jpg temp/1757029231_2640400_1381537359_448736cd1e8954d8580345d4afcfce12_43696339.jpg temp/1757029231_2640400_1381537358_0fab868fd27bb0f08bc292a46ee4a775_54364486.jpg temp/1757029231_2640400_1381537357_86c3e2a048e79f64fc24b88609e6edf7_5302682.jpg temp/1757029231_2640400_1381537356_fb76d4c7fc7bca2f0c776fad46ee6052_26966713.jpg temp/1757029231_2640400_1381537355_bfe820bf7b0caffc7905eb2b6437b1b2_8874297.jpg temp/1757029231_2640400_1381537354_c30b7eac2b63f339e21880469e1a2d2d_20782157.jpg temp/1757029231_2640400_1381537348_e411e13fc78afa59d08ad8820cb880f8_5562686.jpg temp/1757029231_2640400_1381537347_819819e32130d745ce0d63eb12da4e5e_55927551.jpg temp/1757029231_2640400_1381537346_1a3b70090eb4ed9b5eaf0b9943d3713b_87413090.jpg temp/1757029231_2640400_1381537344_ce1aa0d84689f6e45982020cc353da87_83974434.jpg temp/1757029231_2640400_1381537343_c9a6b962440da54aead203177ce879b4_29298157.jpg temp/1757029231_2640400_1381537342_13ee2ed807b4838b7ad9d1ba13747851_11832819.jpg temp/1757029231_2640400_1381537333_65d4746fe203c3c9f22d0c2882fae8f3_88445106.jpg temp/1757029231_2640400_1381537332_4fe74d0d61cd1924ddab83a5b5a2fc45_58268809.jpg temp/1757029231_2640400_1381537331_b359dff9ead5fe1402bfb20e95dfe27f_4875274.jpg temp/1757029231_2640400_1381537330_d3330aacca287aa6253327350ea23cff_14207059.jpg temp/1757029231_2640400_1381537329_9eddb4909ded8a41a66a3d7fc7a95650_35018838.jpg temp/1757029231_2640400_1381537328_1dc878782bc5e4f0b289ad32a2aa0240_69305161.jpg temp/1757029231_2640400_1381537318_b352bfa049d7ade628703905ec4d322c_70710863.jpg temp/1757029231_2640400_1381537317_4447aa03002f20d2668c63c4c6ea3ccf_75736797.jpg temp/1757029231_2640400_1381537316_f13ad7d26778ff6e367877cfd6573e73_91557082.jpg temp/1757029231_2640400_1381537315_b1dc76076233acc9267f6bbb0addbe8c_42521644.jpg temp/1757029231_2640400_1381537314_bd2cddef03e8dd8dd1c8b5ba9d4c24c1_1310804.jpg temp/1757029231_2640400_1381537313_e3333deaa147ab4e729e62bc403d9d83_96502448.jpg temp/1757029231_2640400_1381537224_6402c4d3525a543ae6d2b0536969fb9f_37203538.jpg temp/1757029231_2640400_1381537223_50e86ea5ea9aa95205b97fb45baf353e_44720247.jpg temp/1757029231_2640400_1381537222_acb44684424f7ec9ce9cf4854a310aee_37949726.jpg temp/1757029231_2640400_1381537221_36f43d78724a7b1f65237ca64d850b9c_85128374.jpg temp/1757029231_2640400_1381537220_bceebe400496b71b3522fcf77f562be3_91070490.jpg temp/1757029231_2640400_1381537219_260af20aec29ed1de10f3a64b9e99526_36202953.jpg temp/1757029231_2640400_1381537171_c1d1c917b0d60f4bfd734cf000f9de9d_3149015.jpg temp/1757029231_2640400_1381537169_442b77c6e09421c08699dce3e53cfc72_94924619.jpg temp/1757029231_2640400_1381537166_4b5b47c6419d18e0b59f10d21beaea4d_83894225.jpg temp/1757029231_2640400_1381537164_7dd309c53e77f04c03bc676fcc8ecaac_40732290.jpg temp/1757029231_2640400_1381537162_d1b12ece48b25699518dc794c112d760_24551997.jpg temp/1757029231_2640400_1381537160_e9f21bafe0c080552f2b2840ae64cad8_73113551.jpg temp/1757029231_2640400_1381537081_0e9f2653f76c654aa89fe1e9d6ffe34d_30552779.jpg temp/1757029231_2640400_1381537079_5bcf9582c14bf3dd9ac1bcb4ed788bf5_23020734.jpg temp/1757029231_2640400_1381537077_b8586aa4b007122cef7708f060166516_33598819.jpg temp/1757029231_2640400_1381537075_0f6efd0719a4939cb8b85656f0d50b64_81524020.jpg temp/1757029231_2640400_1381537073_c318d562d081ae5e18635d38fcd0a200_29807878.jpg temp/1757029231_2640400_1381537071_115c81d02f5e02b01b4045718dd9333e_77942147.jpg temp/1757029231_2640400_1381536981_cd3de31efc60807ab07b92b5743fa22f_19710741.jpg temp/1757029231_2640400_1381536979_21d10edc448aabc27cb9844e18ba2bda_41561026.jpg temp/1757029231_2640400_1381536977_11d173331763a0066048ac664d3cb1ee_27919767.jpg temp/1757029231_2640400_1381536975_3affb503eb166a0bb8e2622bedae7c8f_48171006.jpg temp/1757029231_2640400_1381536972_18c27cb51ed64a22ac616b9901df1c6f_53440730.jpg temp/1757029231_2640400_1381536968_74404792452520282f37234f1136234f_1221950.jpg temp/1757029231_2640400_1381536925_58657ac0c91048d74177787e7c6b42b9_53032278.jpg temp/1757029231_2640400_1381536924_9123691af69f1003eb56373808008895_96883444.jpg temp/1757029231_2640400_1381536923_920a5f38e4b9c02af3de91ccaaa395e4_59267743.jpg temp/1757029231_2640400_1381536922_22b3855160609af71ee1625f38274751_20038955.jpg temp/1757029231_2640400_1381536921_ebc2efe01eea66de474861840380e109_75256389.jpg temp/1757029231_2640400_1381536920_f7afd5d095354e435d8adf27593f2bc6_71060676.jpg temp/1757029231_2640400_1381536884_b78bd9d7a2ccbbe0e7c62f1fa2fa7a19_93217852.jpg temp/1757029231_2640400_1381536883_50fb0703244c4a9846c317a4f3635581_89540564.jpg temp/1757029231_2640400_1381536881_16826e36a86f7019c738bccb44336c81_65763733.jpg temp/1757029231_2640400_1381536879_08a10a85fd980e22decdfbfe812afcd5_87834652.jpg temp/1757029231_2640400_1381536877_41ce2f5e6773a9f91614290e3b7277a5_92047596.jpg temp/1757029231_2640400_1381536876_d6e16c0ff142a9a09dcf6dfd4370bb3a_2810121.jpg temp/1757029231_2640400_1381536822_a16a93842999b06c89f903b0f3767f37_1567586.jpg temp/1757029231_2640400_1381536820_073a68a3d3e6a847decb5f546fcb08ff_76151900.jpg temp/1757029231_2640400_1381536818_48927e326e50ca64cc850857f896dd53_48405675.jpg temp/1757029231_2640400_1381536816_84f720c1a68843079d350a2a36088832_6370178.jpg temp/1757029231_2640400_1381536814_f4f1ef3933f8f98af013d5715634dd58_68744733.jpg temp/1757029231_2640400_1381536812_47c239e7e22f2279d3e32bf8fe19ef48_11569846.jpg temp/1757029231_2640400_1381536797_0083ad9079600359c4618b68b8be06a7_78643731.jpg temp/1757029231_2640400_1381536796_ed07fea6b16314d55590613853ca429b_30215387.jpg temp/1757029231_2640400_1381536795_4119d22c73e84cde133235e74127b8b0_2854349.jpg temp/1757029231_2640400_1381536794_ebd3c8a8044b07a87b04bff4b6ffacc7_25151779.jpg temp/1757029231_2640400_1381536793_ef857d0ab1498369d05b89034d1eaeb6_61321015.jpg temp/1757029231_2640400_1381536792_a049b9ca09f702139b04fd414fecbe18_35773549.jpg temp/1757029231_2640400_1381536765_8f2ac1fadfd6f62ce72a611adcb13b44_31565853.jpg temp/1757029231_2640400_1381536764_1a8a60988a85ea67bc86f75be49d8b14_27298624.jpg temp/1757029231_2640400_1381536762_f3903b63796e79fafd98bf3b81e0db58_63541688.jpg temp/1757029231_2640400_1381536761_9ce14bcadb0b6f9add55c2c1c437a854_24461373.jpg temp/1757029231_2640400_1381536760_0d83a38fec81136ec7fb2ef94e8eddef_37531576.jpg temp/1757029231_2640400_1381536757_a52cfd9875ce2e8c286d608f9ee37f80_37026211.jpg temp/1757029231_2640400_1381536744_92960ee3ba5511d155ca0abca025109e_77048737.jpg temp/1757029231_2640400_1381536743_c65e629130f8cd9d931c14fa3039e00e_55578071.jpg temp/1757029231_2640400_1381536742_03cf5ff42e17a07fcd78041f04c48f5f_15559900.jpg temp/1757029231_2640400_1381536741_b83a423938f41d7a4622a09afa6f50cb_69198249.jpg temp/1757029231_2640400_1381536740_6e96f9289f88a184411a15775095aa34_84490204.jpg temp/1757029231_2640400_1381536739_010755be0614b47256933d7af3966141_73861744.jpg temp/1757029231_2640400_1381536607_3e8fd0753b8eafd90522771fe738108c_79258948.jpg temp/1757029231_2640400_1381536605_17b655b533d9e09c8bf07d98ecff4016_16029084.jpg temp/1757029231_2640400_1381536603_afaef8d21bcb14fd2beaabf54edabbb2_99554484.jpg temp/1757029231_2640400_1381536601_857466fc06a01c5bfcfc634198bead54_57496455.jpg temp/1757029231_2640400_1381536598_90f800bdd2f9726edddb144edba8d8ae_72566997.jpg temp/1757029231_2640400_1381536596_5dd42bcd7e522bfd64d3a2296418ab94_12092555.jpg temp/1757029231_2640400_1381536500_237f51b9395dfc49dad82a3b3df57d2a_58122095.jpg temp/1757029231_2640400_1381536498_24ffb2e5f8fc74075b061c95f2ddf777_45919055.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! cuDNN launch failure : input shape([1,2048,7,7]) filter shape([1,1,2048,512]) [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/res5b_branch2a/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1381540573, 1381540569, 1381540565, 1381540561, 1381540557, 1381540553, 1381540328, 1381540324, 1381540320, 1381540316, 1381540312, 1381540308, 1381540118, 1381540115, 1381540112, 1381540109, 1381540106, 1381540103, 1381539716, 1381539712, 1381539708, 1381539704, 1381539700, 1381539696, 1381539355, 1381539350, 1381539344, 1381539339, 1381539334, 1381539329, 1381539034, 1381539030, 1381539023, 1381539018, 1381539013, 1381539008, 1381538775, 1381538770, 1381538765, 1381538760, 1381538755, 1381538750, 1381538462, 1381538458, 1381538454, 1381538450, 1381538446, 1381538442, 1381538046, 1381538041, 1381538037, 1381538034, 1381538031, 1381538027, 1381537921, 1381537918, 1381537915, 1381537913, 1381537911, 1381537909, 1381537804, 1381537802, 1381537800, 1381537798, 1381537796, 1381537794, 1381537711, 1381537709, 1381537707, 1381537705, 1381537702, 1381537699, 1381537610, 1381537607, 1381537604, 1381537601, 1381537598, 1381537595, 1381537550, 1381537549, 1381537548, 1381537547, 1381537546, 1381537545, 1381537540, 1381537539, 1381537538, 1381537537, 1381537536, 1381537535, 1381537507, 1381537506, 1381537505, 1381537504, 1381537503, 1381537502, 1381537447, 1381537446, 1381537445, 1381537444, 1381537443, 1381537442, 1381537370, 1381537369, 1381537368, 1381537367, 1381537366, 1381537365, 1381537359, 1381537358, 1381537357, 1381537356, 1381537355, 1381537354, 1381537348, 1381537347, 1381537346, 1381537344, 1381537343, 1381537342, 1381537333, 1381537332, 1381537331, 1381537330, 1381537329, 1381537328, 1381537318, 1381537317, 1381537316, 1381537315, 1381537314, 1381537313, 1381537224, 1381537223, 1381537222, 1381537221, 1381537220, 1381537219, 1381537171, 1381537169, 1381537166, 1381537164, 1381537162, 1381537160, 1381537081, 1381537079, 1381537077, 1381537075, 1381537073, 1381537071, 1381536981, 1381536979, 1381536977, 1381536975, 1381536972, 1381536968, 1381536925, 1381536924, 1381536923, 1381536922, 1381536921, 1381536920, 1381536884, 1381536883, 1381536881, 1381536879, 1381536877, 1381536876, 1381536822, 1381536820, 1381536818, 1381536816, 1381536814, 1381536812, 1381536797, 1381536796, 1381536795, 1381536794, 1381536793, 1381536792, 1381536765, 1381536764, 1381536762, 1381536761, 1381536760, 1381536757, 1381536744, 1381536743, 1381536742, 1381536741, 1381536740, 1381536739, 1381536607, 1381536605, 1381536603, 1381536601, 1381536598, 1381536596, 1381536500, 1381536498] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.6615269184112549 save_final ERROR in last step tfhub_classification2, cuDNN launch failure : input shape([1,2048,7,7]) filter shape([1,1,2048,512]) [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/res5b_branch2a/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 15.072084188461304 time spend to save output : 0.6674139499664307 total time spend for step 0 : 15.739498138427734 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.48user 4.78system 0:56.29elapsed 34%CPU (0avgtext+0avgdata 2700216maxresident)k 2384inputs+350992outputs (2major+967780minor)pagefaults 0swaps