python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2509065 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 621>200 please execute split_portfolio.py -i 21104060 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2641729'] with mtr_portfolio_ids : ['21104060'] and first list_photo_ids : [] new path : /proc/2509065/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 146 ; length of list_pids : 146 ; length of list_args : 146 time to download the photos : 18.074361562728882 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu Mar 6 17:10:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-03-06 17:10:54.391013: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-06 17:10:54.421964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-06 17:10:54.422273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-06 17:10:54.424438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-06 17:10:54.426624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-06 17:10:54.427166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-06 17:10:54.429742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-06 17:10:54.431288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-06 17:10:54.437132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-06 17:10:54.439019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-06 17:10:54.439465: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-06 17:10:54.467186: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-06 17:10:54.469470: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbf44000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-06 17:10:54.469543: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-06 17:10:54.589860: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1f3f0c50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-06 17:10:54.589917: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-06 17:10:54.591107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-06 17:10:54.591222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-06 17:10:54.591252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-06 17:10:54.591278: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-06 17:10:54.591305: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-06 17:10:54.591332: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-06 17:10:54.591358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-06 17:10:54.591386: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-06 17:10:54.593032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-06 17:10:54.593128: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-06 17:10:54.594088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-06 17:10:54.594109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-06 17:10:54.594120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-06 17:10:54.596372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 9794 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.574610471725464 time used to load_weights : 0.24350500106811523 0it [00:00, ?it/s]temp/1741277434_2509065_1341265227_514548244d6763acbddded7330cbc3b0.jpg temp/1741277434_2509065_1341265222_0e1f5c9749bcb3aa22e6144b68048eb6.jpg temp/1741277434_2509065_1341265216_55b0c95d68488d29e7b044b933ec4a11.jpg temp/1741277434_2509065_1341265213_c57d30c3308fbb0e53692759b4fa6e39.jpg temp/1741277434_2509065_1341265203_8f7b12f53cc8e89045b84c30df225ace.jpg temp/1741277434_2509065_1341265199_011154d4147c1b50ba7cbc325c29ba37.jpg temp/1741277434_2509065_1341265171_320c6eb1d62388506c72378a389232f4.jpg temp/1741277434_2509065_1341265162_82b37b739203b0f884adecb6eb42adce.jpg temp/1741277434_2509065_1341265157_eb08e0f9f0ba645b4db4d02863b05ac1.jpg temp/1741277434_2509065_1341265154_6208f6a6ec5b2d2deadb7ca02924c5a5.jpg temp/1741277434_2509065_1341265151_883c12caa191a4ca7bb3688c6a3c1aa3.jpg temp/1741277434_2509065_1341265148_2b2349a7bf3d1bd902a93c070fc1d2f6.jpg temp/1741277434_2509065_1341265117_8a9bddb3cdaff8efc9c319028001cf88.jpg temp/1741277434_2509065_1341265105_8d4ea55f602505f42f946a5b424590d7.jpg temp/1741277434_2509065_1341265088_da96735d5b528aa2bd9371e57e7b1e7c.jpg temp/1741277434_2509065_1341265081_20c841906f0e74f66ed441ce45c7f29f.jpg temp/1741277434_2509065_1341265076_5d92a4c09bdb50c670a6bf45de532779.jpg temp/1741277434_2509065_1341265070_aac424e0fec99d87c4ca050ab3e99e69.jpg temp/1741277434_2509065_1341265051_b5e3e830c80372e0b17903f7dfc8d29e.jpg temp/1741277434_2509065_1341265042_c6e8250425277d16e6a53ecc7414ea61.jpg temp/1741277434_2509065_1341265035_100a913a0663ac52d57e4bee87b3013a.jpg temp/1741277434_2509065_1341265031_7f0fca20ec864b42f41b3d134eebb06c.jpg temp/1741277434_2509065_1341265013_b50dd49d5de1b81aadbeb4712d624803.jpg temp/1741277434_2509065_1341265009_ade517309cecda8ae3b0d3fe0a4384d8.jpg temp/1741277434_2509065_1341264988_de48ef73463f1b860da9d02e9870d95b.jpg temp/1741277434_2509065_1341264982_b8df3921a765680ddbcbf31eae02a475.jpg temp/1741277434_2509065_1341264977_76ba850006838310fa305a7f0470aa5b.jpg temp/1741277434_2509065_1341264971_194146b32bb3cac7ed4833bd71d33b24.jpg temp/1741277434_2509065_1341264967_ce154ec08711ae7b64d2a494d61de44e.jpg temp/1741277434_2509065_1341264960_6fd88feb5c2b5e535c39f06922977615.jpg temp/1741277434_2509065_1341264943_14350c082654db7cb2391789edd62876.jpg temp/1741277434_2509065_1341264934_3474c8d9fc05264ed36bed0bf03f53c4.jpg temp/1741277434_2509065_1341264927_b8873c080e3721d2d20029729eecef2e.jpg temp/1741277434_2509065_1341264920_0a94cefd09111259005550cadaafa38b.jpg temp/1741277434_2509065_1341264913_328b417af524e047b6c363c5f3f5c7bb.jpg temp/1741277434_2509065_1341264907_f31a70f00dbe0d4a5d0c624c77f63973.jpg temp/1741277434_2509065_1341264883_1a0fcf93a087e97b90705cfaf8676ac1.jpg temp/1741277434_2509065_1341264877_5801a232ae29e60475421ec2d931091d.jpg temp/1741277434_2509065_1341264873_d5590841295b07056d8388be21e71e79.jpg temp/1741277434_2509065_1341264867_0e81584adc8210ba5f7b0c7fc0206f1c.jpg temp/1741277434_2509065_1341264865_c0b8558785c53bc533f974f453cfd67d.jpg temp/1741277434_2509065_1341264859_5a7572033fdfc51de2873bbe37ac506b.jpg temp/1741277434_2509065_1341264835_e0bb41c0163ddfc7dc0733e4f0c25948.jpg temp/1741277434_2509065_1341264827_b9e279259beac13015a9a94aa80777a3.jpg temp/1741277434_2509065_1341264812_373ce01e1ef7bc3ab7d3e132941d21ff.jpg temp/1741277434_2509065_1341264807_518729c758e1ecd9e16cc65cf5658263.jpg temp/1741277434_2509065_1341264805_71c9aff5706cffe72e67a5b560006d1d.jpg temp/1741277434_2509065_1341264802_25c570cb30368f849d6edc8252fa84b3.jpg temp/1741277434_2509065_1341264785_c34cb023a0efc426f42f137e6b7a89ca.jpg temp/1741277434_2509065_1341264775_91af7f7e2d2a067f25a641004d3353d5.jpg temp/1741277434_2509065_1341264768_1c9d135adb12a26922a7a5e17bdee202.jpg temp/1741277434_2509065_1341264760_d85660dd4d790bfccf2f3ec37a8a6570.jpg temp/1741277434_2509065_1341264756_68fab506377bdab500e13a45a6daf4ab.jpg temp/1741277434_2509065_1341264752_f46b712891abca3f9a460773dcde98e8.jpg temp/1741277434_2509065_1341264733_abc7aefb2e642cdd1584fe450ea7d3a5.jpg temp/1741277434_2509065_1341264727_3e61c1688bdb398d2bcaad5e52b9d35f.jpg temp/1741277434_2509065_1341264721_79e5d77d63cbb6f1a30313848b0ad21d.jpg temp/1741277434_2509065_1341264716_734dadea42b016383278440723a2ea54.jpg temp/1741277434_2509065_1341264711_f9a21f3c32985131e579317dd6054a65.jpg temp/1741277434_2509065_1341264706_400cbff58d9b50ba015d30952501e0fd.jpg temp/1741277434_2509065_1341264678_0e5fbbe382f12c1e8d98a32ecb6b48f9.jpg temp/1741277434_2509065_1341264671_f625e8d80bd2ccc88c4ea7ec12e25117.jpg temp/1741277434_2509065_1341264665_7cd680b35d75e8ace534a883a83b9657.jpg temp/1741277434_2509065_1341264662_f4992341a9c1494294a27f6082a9f396.jpg temp/1741277434_2509065_1341264658_0d090fb60822b69532358cecb833c834.jpg temp/1741277434_2509065_1341264653_62e572f522ae70d37e1498cea8f4ce6c.jpg temp/1741277434_2509065_1341264633_55994647331a4ca822c00ecf46ac4aaa.jpg temp/1741277434_2509065_1341264627_01f13a23e5696ca0d0492126bc96d0ce.jpg temp/1741277434_2509065_1341264620_faf9c328b5f2e453312ea2bfe2e3b90d.jpg temp/1741277434_2509065_1341264614_cca2b341e88a64383266bc5ff81257ec.jpg temp/1741277434_2509065_1341264606_88df0708eb566ab967f762e47bbde5f7.jpg temp/1741277434_2509065_1341264597_755e50399344d9884ce5688620df80ee.jpg temp/1741277434_2509065_1341264581_4d41f25b007a9a117bf666d984733acd.jpg temp/1741277434_2509065_1341264576_5d62f6a6f225b59f6f744446dfaef85b.jpg temp/1741277434_2509065_1341264570_9bdae87d88fe0bfbae5206dfbf9c59ae.jpg temp/1741277434_2509065_1341264567_30c58ef12951faae38d0964fe0c9b7e5.jpg temp/1741277434_2509065_1341264562_5c5c042001cb4d9f2f2f9038dd437191.jpg temp/1741277434_2509065_1341264555_4a5d44e1f75ffbff779ea63caadc43b0.jpg temp/1741277434_2509065_1341264517_f8fad375c4b8e3f16abfdf045ce175b1.jpg temp/1741277434_2509065_1341264509_6b9f0361f6d5f95f210d3e4fe976bcce.jpg temp/1741277434_2509065_1341264508_e4c00978257adccfff8014a9cc104313.jpg temp/1741277434_2509065_1341264507_53e5d762d4c41c3087760e2f34654a7c.jpg temp/1741277434_2509065_1341264505_8c9f9408a13d5dd8fc838b4d2f2ba436.jpg temp/1741277434_2509065_1341264503_6f24c3198b5704fb092b31ea0a0ec410.jpg temp/1741277434_2509065_1341264496_28af5f7b0804a3817728c31c78f1657b.jpg temp/1741277434_2509065_1341264494_60f97d9e668e7abec967cbf9d88d3330.jpg temp/1741277434_2509065_1341264492_3ae587ddc4df2b681c7f79183636110f.jpg temp/1741277434_2509065_1341264489_c1524045718d90139ca0f4eb57e3c988.jpg temp/1741277434_2509065_1341264488_94a30bd92913f193cb4f0fb2c2ba8e42.jpg temp/1741277434_2509065_1341264486_09559e0b474534c6d29469a0a087d7a3.jpg temp/1741277434_2509065_1341264484_645be01291f6d7229562d93cf40adb3a.jpg temp/1741277434_2509065_1341264483_b5a3168a5406fd47db9016f783a22d98.jpg temp/1741277434_2509065_1341264481_7b0c733c5a3db733295fc6594a473bcf.jpg temp/1741277434_2509065_1341264479_6cfee641f0300fb42d55c88ae96888c6.jpg temp/1741277434_2509065_1341264477_d0409dbcf6cb92b7a09ef653f427a33d.jpg temp/1741277434_2509065_1341264475_9785d38d7332e832c271090d02f830b7.jpg temp/1741277434_2509065_1341264459_37287b3bc10eeb1cb573d123e74856f6.jpg temp/1741277434_2509065_1341264455_dc5139fc0a088fa5dd842e732b2ca1a8.jpg temp/1741277434_2509065_1341264411_bdbdf7853aed8e9516d757b4d3089286.jpg temp/1741277434_2509065_1341264410_ebf0d46f123e8d6fb3df57807014361a.jpg temp/1741277434_2509065_1341264368_8dde54d8ba7721be5a8857c2b9ac9686.jpg temp/1741277434_2509065_1341264364_55aba62143fd1117e521de19837f0ffd.jpg temp/1741277434_2509065_1341264319_efa40969b5510107299ecaf3bb382e3f.jpg temp/1741277434_2509065_1341264316_6f65db9a7dd0e48715cc614af32ee0ac.jpg temp/1741277434_2509065_1341264274_6e8495335bbbd2d576996bb9e08d5c36.jpg temp/1741277434_2509065_1341264270_c89d7331185f321df66f455bc861ea87.jpg temp/1741277434_2509065_1341264224_35af0d16718c5c8692816fc29816b5e8.jpg temp/1741277434_2509065_1341264222_dfc8448e369d96a219ca7b1a7ab0ebbb.jpg temp/1741277434_2509065_1341264176_71ea3f23fa03c401e75eac6218e3cef9.jpg temp/1741277434_2509065_1341264173_dbc5308c6e89fa5eec66252a818622cf.jpg temp/1741277434_2509065_1341264118_49b43028824b8d15e5c6c96ed600d214.jpg temp/1741277434_2509065_1341264106_2f160116376ae3ec1102f2da8caeaf21.jpg temp/1741277434_2509065_1341264084_24ade47c22d16a74932a3ee79ef7070f.jpg 146it [00:00, 2630.75it/s]2025-03-06 17:11:06.648503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-06 17:11:06.959745: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-06 17:11:07.263714: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-03-06 17:11:07.272869: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1741277434_2509065_1341264081_6a3e01e2cd62cbb85394d17c79c115d1.jpg temp/1741277434_2509065_1341264034_edc124906889ccd1e04b323dfcdd2fc7.jpg temp/1741277434_2509065_1341264030_64c89d30a102ad9af3434cf2fab4a07a.jpg temp/1741277434_2509065_1341263983_9e00fe50402935cb374fc8e033292c86.jpg temp/1741277434_2509065_1341263981_5dc3b29c2d694f19059cd9c3d96e98dc.jpg temp/1741277434_2509065_1341263933_d393c9d3463eb20d39bc78e8ac98892e.jpg temp/1741277434_2509065_1341263929_5d388eb1e3647068708b54a4c824b730.jpg temp/1741277434_2509065_1341263885_8bc877840cc7f84d2f002925a74981be.jpg temp/1741277434_2509065_1341263882_de6d3a168027af2ecf00752038e411a9.jpg temp/1741277434_2509065_1341263836_a9f080c8e5cb6b0c5d0a6027f4693792.jpg temp/1741277434_2509065_1341263793_f2b8de35dddceb1a06ee02209d17ef2c.jpg temp/1741277434_2509065_1341263791_2b2b394aacaef82dd1bb8d6a951bda4e.jpg temp/1741277434_2509065_1341263748_302206637d69de08473b20a218e92368.jpg temp/1741277434_2509065_1341263605_df451c91dcb24b043559335bce5c3ab4.jpg temp/1741277434_2509065_1341263481_7a4dd2951be3df7c71871387493ece50.jpg temp/1741277434_2509065_1341263477_34c97352e5d5e44932261f7b8675228d.jpg temp/1741277434_2509065_1341263429_9c2633992f15fa6067d21e634d35defe.jpg temp/1741277434_2509065_1341263250_2fbd94d9025caf09a2f2c8aae7387f4d.jpg temp/1741277434_2509065_1341263248_fdeabf6b1705e3f0167a5d09c83b9e80.jpg temp/1741277434_2509065_1341263228_2ccdf28d20cb3ff7df36f0ddf89e4781.jpg temp/1741277434_2509065_1341263195_1a486e864b9c0158a749dfacf78e6915.jpg temp/1741277434_2509065_1341263133_3061197eeadfb942131dc646730ee048.jpg temp/1741277434_2509065_1341263128_35d1b122fbcb9ab8b6304e3df234f37c.jpg temp/1741277434_2509065_1341263077_aea8238ea2c9502b27fca7fb6e7e6de0.jpg temp/1741277434_2509065_1341263074_0f21bc299e65e26062435226e4d63df0.jpg temp/1741277434_2509065_1341263035_a2505107958a46ddf6c0c88711295279.jpg temp/1741277434_2509065_1341263032_28370fdbb2ccbed75dcb80e1a5331223.jpg temp/1741277434_2509065_1341262986_9a16a226159a1cea56fd43bba57fcefa.jpg temp/1741277434_2509065_1341262985_86fc965a457b4b369129e7e062b11116.jpg temp/1741277434_2509065_1341262938_f2b13aefaacf7f4fb8829add05b65fba.jpg temp/1741277434_2509065_1341262936_e4ecc58e17177ab9ed74e79240544099.jpg temp/1741277434_2509065_1341262777_5319ce32b9dcef231414eab2248f18e2.jpg temp/1741277434_2509065_1341262760_37c935eeff80a45cf66acef20c240acb.jpg Found 146 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1341265227, 1341265222, 1341265216, 1341265213, 1341265203, 1341265199, 1341265171, 1341265162, 1341265157, 1341265154, 1341265151, 1341265148, 1341265117, 1341265105, 1341265088, 1341265081, 1341265076, 1341265070, 1341265051, 1341265042, 1341265035, 1341265031, 1341265013, 1341265009, 1341264988, 1341264982, 1341264977, 1341264971, 1341264967, 1341264960, 1341264943, 1341264934, 1341264927, 1341264920, 1341264913, 1341264907, 1341264883, 1341264877, 1341264873, 1341264867, 1341264865, 1341264859, 1341264835, 1341264827, 1341264812, 1341264807, 1341264805, 1341264802, 1341264785, 1341264775, 1341264768, 1341264760, 1341264756, 1341264752, 1341264733, 1341264727, 1341264721, 1341264716, 1341264711, 1341264706, 1341264678, 1341264671, 1341264665, 1341264662, 1341264658, 1341264653, 1341264633, 1341264627, 1341264620, 1341264614, 1341264606, 1341264597, 1341264581, 1341264576, 1341264570, 1341264567, 1341264562, 1341264555, 1341264517, 1341264509, 1341264508, 1341264507, 1341264505, 1341264503, 1341264496, 1341264494, 1341264492, 1341264489, 1341264488, 1341264486, 1341264484, 1341264483, 1341264481, 1341264479, 1341264477, 1341264475, 1341264459, 1341264455, 1341264411, 1341264410, 1341264368, 1341264364, 1341264319, 1341264316, 1341264274, 1341264270, 1341264224, 1341264222, 1341264176, 1341264173, 1341264118, 1341264106, 1341264084, 1341264081, 1341264034, 1341264030, 1341263983, 1341263981, 1341263933, 1341263929, 1341263885, 1341263882, 1341263836, 1341263793, 1341263791, 1341263748, 1341263605, 1341263481, 1341263477, 1341263429, 1341263250, 1341263248, 1341263228, 1341263195, 1341263133, 1341263128, 1341263077, 1341263074, 1341263035, 1341263032, 1341262986, 1341262985, 1341262938, 1341262936, 1341262777, 1341262760] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 146 time used for this insertion : 0.14481902122497559 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 15.845893621444702 time spend to save output : 0.15659236907958984 total time spend for step 0 : 16.002485990524292 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 15.81user 4.07system 0:43.72elapsed 45%CPU (0avgtext+0avgdata 2048560maxresident)k 91600inputs+137968outputs (43major+714969minor)pagefaults 0swaps