python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3439918 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2418>200 please execute split_portfolio.py -i 21247140 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2656062'] with mtr_portfolio_ids : ['21247140'] and first list_photo_ids : [] new path : /proc/3439918/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 36.86287808418274 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Tue Mar 11 07:01:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-03-11 07:01:16.357414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-11 07:01:16.637441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-11 07:01:16.637728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-11 07:01:16.640931: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-11 07:01:16.643261: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-11 07:01:16.643683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-11 07:01:16.646328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-11 07:01:16.647713: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-11 07:01:16.652956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-11 07:01:16.654209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-11 07:01:16.654660: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-11 07:01:16.683145: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-11 07:01:16.684885: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa93c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-11 07:01:16.684924: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-11 07:01:16.832325: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x8c04050 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-11 07:01:16.832403: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-11 07:01:16.833347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-11 07:01:16.833427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-11 07:01:16.833449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-11 07:01:16.833468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-11 07:01:16.833486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-11 07:01:16.833506: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-11 07:01:16.833525: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-11 07:01:16.833545: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-11 07:01:16.834597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-11 07:01:16.834656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-11 07:01:16.835439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-11 07:01:16.835461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-11 07:01:16.835473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-11 07:01:16.836801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2862 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-03-11 07:01:24.185406: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-11 07:01:24.186107: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.416212797164917 time used to load_weights : 0.18654966354370117 0it [00:00, ?it/s] 78it [00:00, 774.94it/s]temp/1741672831_3439918_1342852531_9b82315f978d74314c47ecff9e8515a3.jpg temp/1741672831_3439918_1342852522_0f6de9516bb66c903572f667505aff9f.jpg temp/1741672831_3439918_1342852517_e8812d27e54791c2cc6edfa626493673.jpg temp/1741672831_3439918_1342852512_c1e5ae942fdca3a9ee060ab04b324b18.jpg temp/1741672831_3439918_1342852500_b88f140d6d7eb933c333201645a967b7.jpg temp/1741672831_3439918_1342852480_2cb7104d400ffb622e4eb33be41f48f0.jpg temp/1741672831_3439918_1342852399_13e68d29fab9115f64fb8314ba16d191.jpg temp/1741672831_3439918_1342852380_6a800a4bfdcc60849e05217d52e6fbee.jpg temp/1741672831_3439918_1342852350_d7c667ab171dcc0152ba324be8e6ed0d.jpg temp/1741672831_3439918_1342852308_7f5ca52fa92aa7e1ab745b4918d8e0f6.jpg temp/1741672831_3439918_1342852263_fc259fbf2e136c24eee9a0b6e106d3f3.jpg temp/1741672831_3439918_1342852219_d90c02e14f84eed15702421006949b49.jpg temp/1741672831_3439918_1342851853_907dbe5269e41e036225b7f13b0fe776.jpg temp/1741672831_3439918_1342851826_229740457cc2fee93a65793ead4877be.jpg temp/1741672831_3439918_1342851777_54c2835aa6ad26cd93f7f1ce2cefaf23.jpg temp/1741672831_3439918_1342851618_2e90cd6c3cc510e1651e20b08efc80df.jpg temp/1741672831_3439918_1342851449_050c6c0a22a336239386b4e8a359afc4.jpg temp/1741672831_3439918_1342851405_e0231d8fc2da3cc63a9b0440cca72a2f.jpg temp/1741672831_3439918_1342851383_5217ed3005b31109e049bc07efe75447.jpg temp/1741672831_3439918_1342851380_fcc3c3a9e85b1b67d764fd5d79aaaad0.jpg temp/1741672831_3439918_1342851375_24e600f146e1bc69cae9c22b12e1930c.jpg temp/1741672831_3439918_1342851370_26c13bc2f6c492759af719368de9c92e.jpg temp/1741672831_3439918_1342851364_738a29cf1076a18f3d2fde441cc10066.jpg temp/1741672831_3439918_1342851305_58465af37a37fb5137c4bc27fc4e73a2.jpg temp/1741672831_3439918_1342850958_603d2725c9a5a2d600bdcb2028f2dbbb.jpg temp/1741672831_3439918_1342850955_79231fa3d6ce44c17f878e621b4907b5.jpg temp/1741672831_3439918_1342850951_d01d98ca614eebcdf72c1c838a0bc282.jpg temp/1741672831_3439918_1342850945_774388968a2e8af1712f013c50bcbeb2.jpg temp/1741672831_3439918_1342850941_0f85f520b3fd0b547b39b3d029187d95.jpg temp/1741672831_3439918_1342850938_c0bb09e302fc1d17532de22fef33086c.jpg temp/1741672831_3439918_1342850854_4e23fd44d950fe8ee7c013de0f056510.jpg temp/1741672831_3439918_1342850769_51902c753f8d74db6c809dbe1a4a3534.jpg temp/1741672831_3439918_1342850725_b6bf7ff3e7b7d6c2b70dfb418bda1ecc.jpg temp/1741672831_3439918_1342850644_c6eb95ae8110080afbe20b60c5619f9a.jpg temp/1741672831_3439918_1342850485_74c01560a6d066262d538fa0b582958f.jpg temp/1741672831_3439918_1342850279_60d32ac35e16d083048d8cc9cad0e03f.jpg temp/1741672831_3439918_1342849664_23e022403ada098b99dbed62f725e5e7.jpg temp/1741672831_3439918_1342849636_5f5aab6c91d5527a172c4f874156c9e0.jpg temp/1741672831_3439918_1342849630_258a69c1887788880e87da9492abdfe5.jpg temp/1741672831_3439918_1342849626_88f2b4d3179eb625407b29e4fbab01b1.jpg temp/1741672831_3439918_1342849623_fe78298572f93d2ddbfc171d9fb9e8b7.jpg temp/1741672831_3439918_1342849620_c856b9bb9b458ab662c6e4c5a4c7d83c.jpg temp/1741672831_3439918_1342849568_68de1738209f5f440cb2afdb4dae52f8.jpg temp/1741672831_3439918_1342849538_eddcd0818d193d8ecef28e7742a64827.jpg temp/1741672831_3439918_1342849508_399466372449f6ccc74e24535e68124a.jpg temp/1741672831_3439918_1342849477_f1c56246a96144dc62b6561281aa861e.jpg temp/1741672831_3439918_1342849444_fb98cfcb1fb54671e7c1ed2e407e9a35.jpg temp/1741672831_3439918_1342849406_3e6d3e641a7444cd5a518ef111968184.jpg temp/1741672831_3439918_1342849295_0f0f3115a424dfbba58990744c807b24.jpg temp/1741672831_3439918_1342849288_02ad98fa836c8f8b58f269d90d510573.jpg temp/1741672831_3439918_1342849285_82278239e38554dc06e806434762e39f.jpg temp/1741672831_3439918_1342849281_0043089ade2bffc348b45da133e52ac9.jpg temp/1741672831_3439918_1342849277_baadf9a0a51131b65d8cf869901ceb0f.jpg temp/1741672831_3439918_1342849274_63a69f25e16c2e5edc157c5d9fe08366.jpg temp/1741672831_3439918_1342849222_55ef3ee8d1a0aac1c04a43a913d3682b.jpg temp/1741672831_3439918_1342849210_67d2112cf8cdd2880988acf5df8dbcff.jpg temp/1741672831_3439918_1342849205_94a85844963192f627b21ff4e7b6cebf.jpg temp/1741672831_3439918_1342849198_c2e330ee55dce5c9c8ed1cf3bee99760.jpg temp/1741672831_3439918_1342849191_2091d9e3935fe930455aa60864a63518.jpg temp/1741672831_3439918_1342849177_e6f1610cefb30ee45069705efd69e484.jpg temp/1741672831_3439918_1342848852_efc1322a3663d887ef7285647c7c6d3d.jpg temp/1741672831_3439918_1342848829_ee9c26369f37b2a09e91a14d859fdc2b.jpg temp/1741672831_3439918_1342848808_e9a5fb49075f6b8e5fbeafd4f27d135f.jpg temp/1741672831_3439918_1342848803_9bf0dffd7fc9e83a90a466cd140d4b5f.jpg temp/1741672831_3439918_1342848797_717157f09f65981161b302909a15d2ef.jpg temp/1741672831_3439918_1342848783_4c4741cbd4d3aecd7201f73b07a0df9e.jpg temp/1741672831_3439918_1342848744_3974506a18c0a2be734378ec2da74e83.jpg temp/1741672831_3439918_1342848728_5cf4b7c1dd6d4901dbd190b1c8e70361.jpg temp/1741672831_3439918_1342848715_0fe835fd4077613cdeeb073997e40cda.jpg temp/1741672831_3439918_1342848708_0b9d965b2bb725e7a74c9c303ac1917a.jpg temp/1741672831_3439918_1342848701_db7283cfac259b522d10fa5ff0eea2d5.jpg temp/1741672831_3439918_1342848696_ba26fa3f0eb0328739089e695ddc03cb.jpg temp/1741672831_3439918_1342848663_a6ecdbad68ea04062ba2f835d3c70c2c.jpg temp/1741672831_3439918_1342848654_963996df0bb91cfe812e9c91993cce9e.jpg temp/1741672831_3439918_1342848645_af100b19edd85f13adc00ebd09b363c8.jpg temp/1741672831_3439918_1342848640_5071ffa146a5e3cca2c3f88cb8769f1b.jpg temp/1741672831_3439918_1342848635_f42a522b728f27c9dbb2d32dee796db7.jpg temp/1741672831_3439918_1342848630_3b8438aaecb012de2f1f1f61341d03d7.jpg temp/1741672831_3439918_1342848613_b0fa37e52a11b3a6e0d2c7eac05463c7.jpg temp/1741672831_3439918_1342848610_9f287a45cf8079dc2db6649b7c1301c8.jpg temp/1741672831_3439918_1342848607_73b2aa9348a8c622809b121533c91a33.jpg temp/1741672831_3439918_1342848599_c2debd67dd6cbdb434954d5499cdd393.jpg temp/1741672831_3439918_1342848588_77f8c5d5fe2fa341c0fae252cf1d356b.jpg temp/1741672831_3439918_1342848580_735171aefb95a56230dea9214ced5e2f.jpg temp/1741672831_3439918_1342848557_68f5a87ce5afce7e71212e1b6870a826.jpg temp/1741672831_3439918_1342848553_271f1c49579f6ccec2db31622a813994.jpg temp/1741672831_3439918_1342848548_741315069bd39dc567571b36973e1d86.jpg temp/1741672831_3439918_1342848543_544447da0cd2d51845086f235eb22660.jpg temp/1741672831_3439918_1342848539_8bf56f59ad44cd6eccc748fc302f90ae.jpg temp/1741672831_3439918_1342848534_a9933c1656f9115441c0c735b0e967a0.jpg temp/1741672831_3439918_1342848448_d14bd807871971b84d06f75a781a84b3.jpg temp/1741672831_3439918_1342848417_fd2972881b09b6f66ce94aad9958ed81.jpg temp/1741672831_3439918_1342848368_0c3469c296cb99d8c1ca11972ebe2903.jpg temp/1741672831_3439918_1342848314_a2f7333bab17957a90b291f9a783299f.jpg temp/1741672831_3439918_1342848264_c0a3492ebb3f08eb2f51c7bf83c1bf8c.jpg temp/1741672831_3439918_1342848218_da0de6f741c77d5ab294ce9dd0cc2d68.jpg temp/1741672831_3439918_1342847976_49ecca43ac3086b6586839f1e93c1237.jpg temp/1741672831_3439918_1342847972_5a60acbadac3c6ad6710d2cabce98731.jpg temp/1741672831_3439918_1342847946_9fc9717e70a81cbff7289b03562b55a2.jpg temp/1741672831_3439918_1342847920_29897b4e8d55b48873c5ccbb793fef1e.jpg temp/1741672831_3439918_1342847891_0d5276a18fded7d65a848f02e6f4ddf4.jpg temp/1741672831_3439918_1342847871_591cf49e57914e456237271eff9c229b.jpg temp/1741672831_3439918_1342847814_0cf3245e5a0d48cefa9a91cff7fbf852.jpg temp/1741672831_3439918_1342847778_63d98189026fdffd4834d0f1998c3bd9.jpg temp/1741672831_3439918_1342847738_fb3d9c92003455ae25209e772328ba4a.jpg temp/1741672831_3439918_1342847699_97eb4859a34a569efcea31601197641e.jpg temp/1741672831_3439918_1342847670_889bd54849df3421422b27017ecbebd1.jpg temp/1741672831_3439918_1342847630_b7eeab3a261ed9c6032fc44c3bd8df32.jpg temp/1741672831_3439918_1342847336_310c9ce254bf29eacccd726a7373e9dd.jpg temp/1741672831_3439918_1342847256_29058b3886f7f313a36ad327cd99651f.jpg temp/1741672831_3439918_1342847230_28bf363d8039e05452739ba9f3bbcc24.jpg temp/1741672831_3439918_1342847196_f31ab3711b65895cefe95359a8dfa955.jpg temp/1741672831_3439918_1342847170_83cf441585c5dd0eda2c8045d84872f7.jpg 200it [00:00, 1141.30it/s]2025-03-11 07:01:28.508968: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-11 07:01:28.612798: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-03-11 07:01:28.643073: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-11 07:01:28.645662: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-03-11 07:01:28.646986: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1741672831_3439918_1342847164_83b4291ca9b13bb9dcad2a03c140bb34.jpg temp/1741672831_3439918_1342847121_f9e3e4a4c704194ad5625160b20d1e77.jpg temp/1741672831_3439918_1342847115_ee076a6b9f0766e733cb23f9818e8e42.jpg temp/1741672831_3439918_1342847110_236e7a292abbeb2d571a53f9c5e1d508.jpg temp/1741672831_3439918_1342847098_55a46deaecf973106bff1610d299a521.jpg temp/1741672831_3439918_1342847080_a7ccc4ef0fb4572d66422b2c5cefd2ad.jpg temp/1741672831_3439918_1342847077_ee4435fa3e570b770e7ba577f80329f1.jpg temp/1741672831_3439918_1342847066_118e1fcdaf55a05b95c770cfa5823ad8.jpg temp/1741672831_3439918_1342847062_4a007ffe515acc17288b2172930dc444.jpg temp/1741672831_3439918_1342847056_2476d96419571a3be796dfdd0f13c79e.jpg temp/1741672831_3439918_1342847051_f21ec67b1ddf3fc9ef969de5d6ef8695.jpg temp/1741672831_3439918_1342847025_42ab24441649373fcaa6457ad15e101d.jpg temp/1741672831_3439918_1342847021_ce0af820e755691cb5d711bec912a6ad.jpg temp/1741672831_3439918_1342847007_dcb96d9c8f309f0eefc7c4dec14184f4.jpg temp/1741672831_3439918_1342847004_d15761c86088e2b21f0fba817e0b6d93.jpg temp/1741672831_3439918_1342847000_d820d61a85463c38fc448ed077587555.jpg temp/1741672831_3439918_1342846994_e74e782d8824cd01d6cf92f4f22cf5bd.jpg temp/1741672831_3439918_1342846991_2ee6e5f05b49ddd98cc656a60f9dc7e9.jpg temp/1741672831_3439918_1342846984_928a2f31eef4a1cb8c7d51603184df62.jpg temp/1741672831_3439918_1342846896_ccb44d0f158bc3099853184382583fbc.jpg temp/1741672831_3439918_1342846892_146b871dfaf382332c62a069bb5597f5.jpg temp/1741672831_3439918_1342846890_af4ce9606d7420da17d6d1042e926a8c.jpg temp/1741672831_3439918_1342846888_40023f18b3e6923e921737d20f6dfadd.jpg temp/1741672831_3439918_1342846885_e408961b7910e80825162186b149b2ac.jpg temp/1741672831_3439918_1342846881_cd4881d1469490d83e66259b8f8c63cc.jpg temp/1741672831_3439918_1342846838_add660b4356fae2787d24d52dcaa704b.jpg temp/1741672831_3439918_1342846834_8ab77d0807a0b3cdbdc7237f931c62c2.jpg temp/1741672831_3439918_1342846804_ed24f2476435175605695d955214a024.jpg temp/1741672831_3439918_1342846797_172e1942479011d010aa0fc28be785fb.jpg temp/1741672831_3439918_1342846776_6c5526c64938e55eda600b30db282c6a.jpg temp/1741672831_3439918_1342846771_fdc8cf5d5b044f0cba8e469afb99e35b.jpg temp/1741672831_3439918_1342846602_6eb79ecc93856118f506ad117eb25f56.jpg temp/1741672831_3439918_1342846598_971ea481219bf4f5f47d5021db9c568a.jpg temp/1741672831_3439918_1342846578_3852281951ea075cb8a0fa85d6869724.jpg temp/1741672831_3439918_1342846554_19510f6dcb21724f37816b80662dd4f0.jpg temp/1741672831_3439918_1342846522_980bc919695fe1279cb89749226747c3.jpg temp/1741672831_3439918_1342846482_3c61f8120ff8a7a95e4266b561f1e70d.jpg temp/1741672831_3439918_1342845845_e7c9ba194c7e65306938391e01d45f82.jpg temp/1741672831_3439918_1342845822_c1d2870078cf49c4e3225a6976ef40b4.jpg temp/1741672831_3439918_1342845797_58a0b81b7f1b7e09e5cc9765c94d451c.jpg temp/1741672831_3439918_1342845768_8d119d8de7d3cdbf6400e054e0763757.jpg temp/1741672831_3439918_1342845736_66006c12732f0afdcbbfbe5932fd86e6.jpg temp/1741672831_3439918_1342845717_379fbe7f7237415ab92a078d580a99e5.jpg temp/1741672831_3439918_1342845629_f54691da04cdb14ce1ec20a2da175544.jpg temp/1741672831_3439918_1342845626_147858c2b080a4b08acceab4c48f8b5d.jpg temp/1741672831_3439918_1342845620_96c1e0ceed643b3a3254b1411161bb9f.jpg temp/1741672831_3439918_1342845618_d93e2ff09a5b17fc367941b279c2b38f.jpg temp/1741672831_3439918_1342845610_44594641884d409cbd8dcd383da5bc1e.jpg temp/1741672831_3439918_1342845609_102e959bba7ea29eeea51e49e6bcbe78.jpg temp/1741672831_3439918_1342845142_492c820b70dc1d50d11fbb268562c3de.jpg temp/1741672831_3439918_1342845137_1541eb563c984aaecb878b367a736173.jpg temp/1741672831_3439918_1342845133_6f6547b6ce77ab34a781d3e3d7719165.jpg temp/1741672831_3439918_1342845126_71d19dc57935154d49df33b9329e47e9.jpg temp/1741672831_3439918_1342845120_101571adf20cb7ad33f295a7dbb46327.jpg temp/1741672831_3439918_1342845112_5eaecf3418d2cef544292099527f43a3.jpg temp/1741672831_3439918_1342845047_100cfe11d3d73a73582cba2b05a8101b.jpg temp/1741672831_3439918_1342845042_54bbc109c6527c54e6c8fdf9668fc480.jpg temp/1741672831_3439918_1342845037_a0df7ece63f27e891e4bedba54b1a76a.jpg temp/1741672831_3439918_1342845030_83c70e8515da36b4375526af01663f11.jpg temp/1741672831_3439918_1342845024_46a428f6a0729813bdd3764e1a16ae1d.jpg temp/1741672831_3439918_1342845019_83aec6b9f4dc28d66083b40bfe020dc5.jpg temp/1741672831_3439918_1342844966_0600329225415ff146e7150ba6348b43.jpg temp/1741672831_3439918_1342844949_1c479f8923632d86cdc5a09ff11c7e75.jpg temp/1741672831_3439918_1342844920_9a1373f1e12aacb34a44e5a762696cc5.jpg temp/1741672831_3439918_1342844892_feb146a1ee076b8d61a1d2b92aea3cab.jpg temp/1741672831_3439918_1342844864_039294ef7baf68cfc27a28965401fd63.jpg temp/1741672831_3439918_1342844841_03af45f8f0019ab04b98b5d23cc493b2.jpg temp/1741672831_3439918_1342844609_f21509232e3a2163de8d46feb85d95c8.jpg temp/1741672831_3439918_1342844545_a80d23d5e09eca4d8de5e18beb860c6f.jpg temp/1741672831_3439918_1342844503_f3a897c2406b98d70b9410bfe30fecc2.jpg temp/1741672831_3439918_1342844442_f1b257fadb2cd999c7e35b1059d1ddb0.jpg temp/1741672831_3439918_1342844401_176285f458d3192b64b7c7317d3efc6f.jpg temp/1741672831_3439918_1342844372_9372ba0b05f00ac389af96e2b621bdec.jpg temp/1741672831_3439918_1342844295_1578c6095940b2ec49f04400b624340c.jpg temp/1741672831_3439918_1342844286_66f68697431e49d93564530d931860f5.jpg temp/1741672831_3439918_1342844271_0ba361f8e370edbc18e51db4c06f0e6a.jpg temp/1741672831_3439918_1342844265_1c49ff3f4b47e356e346e0681ad86ebd.jpg temp/1741672831_3439918_1342844258_3f8b403a2663670d425c3218f4013ba5.jpg temp/1741672831_3439918_1342844254_0ec76e7d800e8d5440cfb50e3c6e46f3.jpg temp/1741672831_3439918_1342844208_b796e1d312c4c585d11d130f5d035022.jpg temp/1741672831_3439918_1342844188_babbc8a5bb292379ded82800797b140f.jpg temp/1741672831_3439918_1342844179_4085744ffe5a852b56bc365fd6763af0.jpg temp/1741672831_3439918_1342844167_805fc48327b3198f7b14fd5a8dcf2a12.jpg temp/1741672831_3439918_1342844095_d2e290d749a5ac8f3454fe849db4fc95.jpg temp/1741672831_3439918_1342844010_218216c4e0db42217d0d8b6cdc31d930.jpg temp/1741672831_3439918_1342843852_2d2c01c8150afb460c235f84bf292600.jpg temp/1741672831_3439918_1342843842_a3b546e2948df9db2b87a4c6f650b402.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1342852531, 1342852522, 1342852517, 1342852512, 1342852500, 1342852480, 1342852399, 1342852380, 1342852350, 1342852308, 1342852263, 1342852219, 1342851853, 1342851826, 1342851777, 1342851618, 1342851449, 1342851405, 1342851383, 1342851380, 1342851375, 1342851370, 1342851364, 1342851305, 1342850958, 1342850955, 1342850951, 1342850945, 1342850941, 1342850938, 1342850854, 1342850769, 1342850725, 1342850644, 1342850485, 1342850279, 1342849664, 1342849636, 1342849630, 1342849626, 1342849623, 1342849620, 1342849568, 1342849538, 1342849508, 1342849477, 1342849444, 1342849406, 1342849295, 1342849288, 1342849285, 1342849281, 1342849277, 1342849274, 1342849222, 1342849210, 1342849205, 1342849198, 1342849191, 1342849177, 1342848852, 1342848829, 1342848808, 1342848803, 1342848797, 1342848783, 1342848744, 1342848728, 1342848715, 1342848708, 1342848701, 1342848696, 1342848663, 1342848654, 1342848645, 1342848640, 1342848635, 1342848630, 1342848613, 1342848610, 1342848607, 1342848599, 1342848588, 1342848580, 1342848557, 1342848553, 1342848548, 1342848543, 1342848539, 1342848534, 1342848448, 1342848417, 1342848368, 1342848314, 1342848264, 1342848218, 1342847976, 1342847972, 1342847946, 1342847920, 1342847891, 1342847871, 1342847814, 1342847778, 1342847738, 1342847699, 1342847670, 1342847630, 1342847336, 1342847256, 1342847230, 1342847196, 1342847170, 1342847164, 1342847121, 1342847115, 1342847110, 1342847098, 1342847080, 1342847077, 1342847066, 1342847062, 1342847056, 1342847051, 1342847025, 1342847021, 1342847007, 1342847004, 1342847000, 1342846994, 1342846991, 1342846984, 1342846896, 1342846892, 1342846890, 1342846888, 1342846885, 1342846881, 1342846838, 1342846834, 1342846804, 1342846797, 1342846776, 1342846771, 1342846602, 1342846598, 1342846578, 1342846554, 1342846522, 1342846482, 1342845845, 1342845822, 1342845797, 1342845768, 1342845736, 1342845717, 1342845629, 1342845626, 1342845620, 1342845618, 1342845610, 1342845609, 1342845142, 1342845137, 1342845133, 1342845126, 1342845120, 1342845112, 1342845047, 1342845042, 1342845037, 1342845030, 1342845024, 1342845019, 1342844966, 1342844949, 1342844920, 1342844892, 1342844864, 1342844841, 1342844609, 1342844545, 1342844503, 1342844442, 1342844401, 1342844372, 1342844295, 1342844286, 1342844271, 1342844265, 1342844258, 1342844254, 1342844208, 1342844188, 1342844179, 1342844167, 1342844095, 1342844010, 1342843852, 1342843842] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.10127496719360352 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 20.265003204345703 time spend to save output : 0.11238312721252441 total time spend for step 0 : 20.377386331558228 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.97user 4.42system 1:03.24elapsed 30%CPU (0avgtext+0avgdata 1798220maxresident)k 95288inputs+365216outputs (69major+685220minor)pagefaults 0swaps