python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3575394 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 392>100 please execute split_portfolio.py -i 23084284 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2885854'] with mtr_portfolio_ids : ['23084284'] and first list_photo_ids : [] new path : /proc/3575394/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 12.312674045562744 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat May 17 07:40:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-17 07:40:43.085234: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-17 07:40:43.115559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-17 07:40:43.115806: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 07:40:43.117651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 07:40:43.119483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-17 07:40:43.119793: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-17 07:40:43.121857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-17 07:40:43.122974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-17 07:40:43.127172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 07:40:43.128398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-17 07:40:43.128780: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-17 07:40:43.155793: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-17 07:40:43.157713: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f30ec000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-17 07:40:43.157743: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-17 07:40:43.271745: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x17bce850 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-17 07:40:43.271819: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-17 07:40:43.273425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-17 07:40:43.273530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 07:40:43.273561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 07:40:43.273589: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-17 07:40:43.273615: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-17 07:40:43.273642: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-17 07:40:43.273668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-17 07:40:43.273694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 07:40:43.275479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-17 07:40:43.275587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 07:40:43.276886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-17 07:40:43.276920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-17 07:40:43.276934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-17 07:40:43.278856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 7431 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.213776350021362 time used to load_weights : 0.2070469856262207 0it [00:00, ?it/s] 100it [00:00, 2026.41it/s]2025-05-17 07:40:53.690301: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 07:40:54.033882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 07:40:54.849789: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-17 07:40:54.860895: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1747460428_3575394_1359081087_d65f96fa79ae39fd75050ca6cfa7ea0e.jpg temp/1747460428_3575394_1359081085_20539aee3fad901da81ddb80a3621098.jpg temp/1747460428_3575394_1359081083_d93f9c70e5c8c73a1321fa95b789daaf.jpg temp/1747460428_3575394_1359081080_2f2d340c3e3f97547f0bab9018cffabd.jpg temp/1747460428_3575394_1359081078_2910bb44e37e52128d988b0eb1afaae4.jpg temp/1747460428_3575394_1359081074_fc2baf1cc0d66adb4e23b93b951ee82c.jpg temp/1747460428_3575394_1359081002_973aec4f5459606bb6185daf9ced55f3.jpg temp/1747460428_3575394_1359081000_1c18520db2e75472c31404b171de2b6a.jpg temp/1747460428_3575394_1359080997_512baeaddf74fd10f4bb297b18c4e51c.jpg temp/1747460428_3575394_1359080994_a0f5ebea06daf80bfc970ce9eec8b94e.jpg temp/1747460428_3575394_1359080991_a591184867c84d30b3dc3a4058479665.jpg temp/1747460428_3575394_1359080988_e887f6a991ea254565dbc6c48728008c.jpg temp/1747460428_3575394_1359080806_3b34ca2e32aeba9bdac0592feb32d76b.jpg temp/1747460428_3575394_1359080804_6536a78e43ca351255d6aba0971c10ec.jpg temp/1747460428_3575394_1359080801_d3f05874274b53b3cd5c582282f1e82f.jpg temp/1747460428_3575394_1359080798_3ef481d9a706baa00d1c2472a35e021c.jpg temp/1747460428_3575394_1359080795_95ae998475160b7ab166c13d6f239b69.jpg temp/1747460428_3575394_1359080793_969db5c0cbb94257f4b9c24884a3eeaa.jpg temp/1747460428_3575394_1359080670_bebac6ddeba39d9b87239f6154866806.jpg temp/1747460428_3575394_1359080669_bd181e60a93fe7ba119103c5191c44fa.jpg temp/1747460428_3575394_1359080667_0797e328a421bc944134ba80e74dd753.jpg temp/1747460428_3575394_1359080664_3e74389fd1fb8ba1554c8973b0fcf27f.jpg temp/1747460428_3575394_1359080661_716e2115c4fd3e873ba1054d9725b0db.jpg temp/1747460428_3575394_1359080657_d087c9910131a9ac55937f536d23cfea.jpg temp/1747460428_3575394_1359080566_338e92019931e998a8e622b4b49cb9f5.jpg temp/1747460428_3575394_1359080565_e36815779721cf69be66459a412a34ca.jpg temp/1747460428_3575394_1359080562_5a653ca027507c4d76e36de080d458e7.jpg temp/1747460428_3575394_1359080560_25cb3fa6a9301e0fdf14e420487deed6.jpg temp/1747460428_3575394_1359080559_3335773917741d814db73902dbdbfb4b.jpg temp/1747460428_3575394_1359080557_12293f1925eb2c3a4350875f2371aa86.jpg temp/1747460428_3575394_1359080489_8ca3f3c161dd7db8d5581aad889cff1c.jpg temp/1747460428_3575394_1359080486_df7d3e93fac0fdd211dbb42423c2cb60.jpg temp/1747460428_3575394_1359080484_eea158dbca3bc5303acc59e4c6489396.jpg temp/1747460428_3575394_1359080482_e08109d033113d5a661df37e0072c809.jpg temp/1747460428_3575394_1359080481_60c84201de76f3173b7be810ae37276e.jpg temp/1747460428_3575394_1359080478_84050c77691b579606a4886adc282443.jpg temp/1747460428_3575394_1359080419_15ff84ca1f5a5101ec8d8ad36e422639.jpg temp/1747460428_3575394_1359080417_22ec9ddac31114f74e8b19495b03b207.jpg temp/1747460428_3575394_1359080416_667e2aa17c5c50309a8304cdb46efca7.jpg temp/1747460428_3575394_1359080414_4e567de0c746a7a0e835dfa8170f8297.jpg temp/1747460428_3575394_1359080410_0e011f53cea41ced89072f1ac18b6262.jpg temp/1747460428_3575394_1359080408_0287e405d533effd79c2545f5802729b.jpg temp/1747460428_3575394_1359080359_ef1b51b23a81a11a34cbe135d768dbb5.jpg temp/1747460428_3575394_1359080356_eb45760292be5da737cb71e38fab387a.jpg temp/1747460428_3575394_1359080353_66a9577cda0957efcd43fc48fefe87b8.jpg temp/1747460428_3575394_1359080351_f46819d3e3bc0333bb163d9ca9005452.jpg temp/1747460428_3575394_1359080350_3fc6eba9f363dd8858a855869ef36491.jpg temp/1747460428_3575394_1359080349_b68aa0b654a8e4c955f5b07c6a667ad2.jpg temp/1747460428_3575394_1359080179_a74ff78c5c3fb3986c352c148a356700.jpg temp/1747460428_3575394_1359080177_d01d2f14d1251a9d294ad58864ccaafd.jpg temp/1747460428_3575394_1359080175_c7c0b3d4ce5d20414a1db5717d4b5489.jpg temp/1747460428_3575394_1359080173_dc0d194c4ceea2bb5b11877828e469fa.jpg temp/1747460428_3575394_1359080170_7890fa559599b2c7cdd8125949b08596.jpg temp/1747460428_3575394_1359080167_11dc88325a7bfa73e443e2b534abaace.jpg temp/1747460428_3575394_1359079919_17f49420c25cb6c33a3e820cf6cda6a4.jpg temp/1747460428_3575394_1359079916_6c75ae19c282d38505262ebe3b723b6d.jpg temp/1747460428_3575394_1359079912_1e97331af860077e4e2348da2fdbd0b6.jpg temp/1747460428_3575394_1359079908_f974fb8b3af4f14f19eb5971b34bc126.jpg temp/1747460428_3575394_1359079905_517d5b5fe7530e42cf14b5f36bcfa762.jpg temp/1747460428_3575394_1359079901_838b9f5e03ba0e83571a37b1899d126a.jpg temp/1747460428_3575394_1359079759_f7a10cb1e704af4f099e4a5a564e188d.jpg temp/1747460428_3575394_1359079755_94b97dd678da1e0a29a2de1ea410f687.jpg temp/1747460428_3575394_1359079748_68a9e2f16ec4993f3cc9eb2bf3937053.jpg temp/1747460428_3575394_1359079745_de39cfdd691961b88e35dca895935e4f.jpg temp/1747460428_3575394_1359079742_5af8add3c883787bff1d16b91b304d56.jpg temp/1747460428_3575394_1359079740_5410bc028daf10f0b66b154108bcd401.jpg temp/1747460428_3575394_1359079480_81833d11668ec0fafe60890070ff5266.jpg temp/1747460428_3575394_1359079478_4e752ea64400c9dabdba1b68239a98fd.jpg temp/1747460428_3575394_1359079476_a9124d6dd612198e6e71c676a7da286c.jpg temp/1747460428_3575394_1359079472_ecab85b5fa12f4a5502343177eb83ae5.jpg temp/1747460428_3575394_1359079471_2c58982280194fecccf9651ba63e5f52.jpg temp/1747460428_3575394_1359079468_db044c180509ee7463ced4920035e9ab.jpg temp/1747460428_3575394_1359079343_d8deff2cdbe0ca9a67b25a8bcb372136.jpg temp/1747460428_3575394_1359079341_69d1f97c9ac464ae8428fa85d14c097b.jpg temp/1747460428_3575394_1359079339_08282bcc022425314ac678b4b5fd8122.jpg temp/1747460428_3575394_1359079337_6ab2d3b7122b575a3d190cf78cdc73d7.jpg temp/1747460428_3575394_1359079335_eff366bde2b4c1eab61762cd9d876347.jpg temp/1747460428_3575394_1359079333_f424dcf78924e8e7d08a1df9d656a963.jpg temp/1747460428_3575394_1359079250_7c1c4152d8a47cb97e419491b13f256b.jpg temp/1747460428_3575394_1359079248_34ba7c46ab07350e8ac677fcaa1ca975.jpg temp/1747460428_3575394_1359079246_96cf71290d2b8f820f503471a610d9f2.jpg temp/1747460428_3575394_1359079243_f6eb0932e827001a3b9fd31d7a918b5a.jpg temp/1747460428_3575394_1359079240_0cdfbd70eb01a458ca1dfe739e4849ba.jpg temp/1747460428_3575394_1359079236_90bc8382f703041bcacdbb6e391bf7d2.jpg temp/1747460428_3575394_1359079141_dac7671f235360a863230636b5b83bf0.jpg temp/1747460428_3575394_1359079139_8d102f5162fd9f6a2a0078521480da61.jpg temp/1747460428_3575394_1359079137_5e9cd329556b676e07c526f1eadae735.jpg temp/1747460428_3575394_1359079081_3f61338cd77a9bc6355a062a3a3f310c.jpg temp/1747460428_3575394_1359079079_e9709857f37760524d12cbebd907ba37.jpg temp/1747460428_3575394_1359079075_565f3e4ede87f06d1f47e819dc5cab71.jpg temp/1747460428_3575394_1359079073_4550e3c85b345e74cc5b1fac98ffe265.jpg temp/1747460428_3575394_1359079071_c1673af08f00a7c4a32b5fcf79c746b3.jpg temp/1747460428_3575394_1359079069_4db8bf6a760077728859080c971b75f1.jpg temp/1747460428_3575394_1359078972_7d2c312597fc0e5e71a5d38e1046ded9.jpg temp/1747460428_3575394_1359078969_bc46768d59931f6a02184e4ed8f6acce.jpg temp/1747460428_3575394_1359078967_5d53f803add0656ab3e3a6ca7e3945b7.jpg temp/1747460428_3575394_1359078964_f523e391ce0550dbbddc2843a254415f.jpg temp/1747460428_3575394_1359078882_6a64add5322cf6e8d24eb85645a0ee08.jpg temp/1747460428_3575394_1359078881_e8725185c78a961570cd0785ed894072.jpg temp/1747460428_3575394_1359078878_e68fd00da62fb94d1ae87bf8901cd091.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1359081087, 1359081085, 1359081083, 1359081080, 1359081078, 1359081074, 1359081002, 1359081000, 1359080997, 1359080994, 1359080991, 1359080988, 1359080806, 1359080804, 1359080801, 1359080798, 1359080795, 1359080793, 1359080670, 1359080669, 1359080667, 1359080664, 1359080661, 1359080657, 1359080566, 1359080565, 1359080562, 1359080560, 1359080559, 1359080557, 1359080489, 1359080486, 1359080484, 1359080482, 1359080481, 1359080478, 1359080419, 1359080417, 1359080416, 1359080414, 1359080410, 1359080408, 1359080359, 1359080356, 1359080353, 1359080351, 1359080350, 1359080349, 1359080179, 1359080177, 1359080175, 1359080173, 1359080170, 1359080167, 1359079919, 1359079916, 1359079912, 1359079908, 1359079905, 1359079901, 1359079759, 1359079755, 1359079748, 1359079745, 1359079742, 1359079740, 1359079480, 1359079478, 1359079476, 1359079472, 1359079471, 1359079468, 1359079343, 1359079341, 1359079339, 1359079337, 1359079335, 1359079333, 1359079250, 1359079248, 1359079246, 1359079243, 1359079240, 1359079236, 1359079141, 1359079139, 1359079137, 1359079081, 1359079079, 1359079075, 1359079073, 1359079071, 1359079069, 1359078972, 1359078969, 1359078967, 1359078964, 1359078882, 1359078881, 1359078878] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.3016679286956787 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 14.565001726150513 time spend to save output : 0.30826497077941895 total time spend for step 0 : 14.873266696929932 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.22user 4.45system 0:31.60elapsed 59%CPU (0avgtext+0avgdata 2387012maxresident)k 2112inputs+104248outputs (3major+722927minor)pagefaults 0swaps