python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2794098 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7704>200 please execute split_portfolio.py -i 23317049 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2948044'] with mtr_portfolio_ids : ['23317049'] and first list_photo_ids : [] new path : /proc/2794098/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 35.53165125846863 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Tue May 27 04:11:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-05-27 04:11:11.295800: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-27 04:11:11.327779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 04:11:11.328021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 04:11:11.329784: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 04:11:11.331451: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 04:11:11.331753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 04:11:11.335026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 04:11:11.336358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 04:11:11.341841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 04:11:11.342852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 04:11:11.343216: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-27 04:11:11.367112: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-27 04:11:11.368945: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f5c44000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-27 04:11:11.368979: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-27 04:11:11.475384: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xf272dc0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-27 04:11:11.475439: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-27 04:11:11.476457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 04:11:11.476535: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 04:11:11.476565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 04:11:11.476592: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 04:11:11.476619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 04:11:11.476645: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 04:11:11.476671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 04:11:11.476698: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 04:11:11.478010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 04:11:11.478071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 04:11:11.478889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-27 04:11:11.478929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-27 04:11:11.478943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-27 04:11:11.480311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3033 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-27 04:11:18.151240: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 04:11:18.151899: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.116630792617798 time used to load_weights : 0.21570110321044922 0it [00:00, ?it/s]temp/1748311828_2794098_1361046093_3cd57af8e7f0ec99ec5eca596ad694ef.jpg temp/1748311828_2794098_1361046088_323e4d3d4e55f8ce7c42c5fe2cf483a2.jpg temp/1748311828_2794098_1361046086_0d876f7471d0061ddd10c8b099764cf2.jpg temp/1748311828_2794098_1361046084_a6cb97b3448ce6c21d67dd594a28c992.jpg temp/1748311828_2794098_1361046082_3fa164182995c3a839004e7a5028c785.jpg temp/1748311828_2794098_1361046080_aa80f05799364a5ab281eec9e1c416d8.jpg temp/1748311828_2794098_1361046039_89abf8af8196d1c1d972962e3c704b24.jpg temp/1748311828_2794098_1361046035_5cbaf58b8d59bd58838f2e1dc4dcbd65.jpg temp/1748311828_2794098_1361046031_410f5ce4d95661a8fcb535c50007beb8.jpg temp/1748311828_2794098_1361046026_a7b3ec882c2521efed99f82f3e436a73.jpg temp/1748311828_2794098_1361046021_52abbf44cadaf6c2e5cb47000152396a.jpg temp/1748311828_2794098_1361046018_32851d0840c547b41581a69848e3d0ed.jpg temp/1748311828_2794098_1361046008_aae0d3479877258c3c665b84b373025e.jpg temp/1748311828_2794098_1361046006_621954409bfa49e49885877ffab53bc7.jpg temp/1748311828_2794098_1361046005_ce1c477705ee3b71fca708e3ed0102c8.jpg temp/1748311828_2794098_1361046004_b13621ea14ebc4ecf843a8d727086f69.jpg temp/1748311828_2794098_1361046003_ac2f2be18beddc2c1c251d3b9ebc65d2.jpg temp/1748311828_2794098_1361045999_6dfbb5fc92faa4c762c8d66ff43b6580.jpg temp/1748311828_2794098_1361045938_ebae41a311eff698fe63f84cfd38a13e.jpg temp/1748311828_2794098_1361045934_981f5a4998243882e9e6dee0f44c81a4.jpg temp/1748311828_2794098_1361045929_281b6e6a1af09124886071ce91816ab5.jpg temp/1748311828_2794098_1361045927_46652b846a3181371b3cea1029db84d8.jpg temp/1748311828_2794098_1361045910_6b4cd9255c873a9115880b93d7fa78d4.jpg temp/1748311828_2794098_1361045909_5d63d2c266cfe649880312d3a28d969b.jpg temp/1748311828_2794098_1361045871_401ffebea1fb582048b62e55d7aff050.jpg temp/1748311828_2794098_1361045869_ffe4c1a104ae4297928b670c009fc6be.jpg temp/1748311828_2794098_1361045868_47b531aef8114e1b1381cb45615c27b9.jpg temp/1748311828_2794098_1361045867_ace4a24181aa8e73fe3bdcb97bc618dd.jpg temp/1748311828_2794098_1361045866_7e5f86c4ae6d694853a617ab13e5aab5.jpg temp/1748311828_2794098_1361045865_bfd630347f59c03c27a105eda2dd8155.jpg temp/1748311828_2794098_1361045842_85f42eb9e08bde68b5dc660774baa4b9.jpg temp/1748311828_2794098_1361045838_ba1925fe2dde1b66a781c42950cc4077.jpg temp/1748311828_2794098_1361045836_4904709527aaac7fc985b7550a7b0811.jpg temp/1748311828_2794098_1361045832_4b64d73a5af5b1a90d5d3f83a020dd01.jpg temp/1748311828_2794098_1361045829_25d70390c268b2e68a45c86130c22544.jpg temp/1748311828_2794098_1361045826_5b68b9077079ff164934419b33cf9f30.jpg temp/1748311828_2794098_1361045780_e675b9af10a0a8dfe1a54c461f0c4c39.jpg temp/1748311828_2794098_1361045778_b99eca26000474224ddf06ddf9de30de.jpg temp/1748311828_2794098_1361045775_2e3dd894b74fca6d78a8e77e9ea6a280.jpg temp/1748311828_2794098_1361045772_c04291cb067b44186bcc0953d6c9c916.jpg temp/1748311828_2794098_1361045768_7885db40b3af3912e631e05044b2b090.jpg temp/1748311828_2794098_1361045765_b955ac9aa4df0d38153868b012afb57d.jpg temp/1748311828_2794098_1361045756_8098819923227142f8a236959b44d076.jpg temp/1748311828_2794098_1361045752_67c23d96183897c61b8303aeb02342ea.jpg temp/1748311828_2794098_1361045749_d09e2d06b0df9504ae23fe0fff0c8b42.jpg temp/1748311828_2794098_1361045746_9619b100d1228c431f2d8cd516b6404b.jpg temp/1748311828_2794098_1361045743_457531c34f3585396a3a99b7b6e36f61.jpg temp/1748311828_2794098_1361045740_0006946e7d69c12406d28fc8c1615819.jpg temp/1748311828_2794098_1361045715_bcc4c9971e733bccaaa5faff80679d0c.jpg temp/1748311828_2794098_1361045710_0b074f533acd2fc3ba9e6a129f5c46da.jpg temp/1748311828_2794098_1361045705_6dc55ef68fa7f61b89526bc53f93cdde.jpg temp/1748311828_2794098_1361045701_4cfe1bb8b6ba298ce8a7c1a76f92c946.jpg temp/1748311828_2794098_1361045696_5067cea6f0b60f7e35edb010f7bf7c03.jpg temp/1748311828_2794098_1361045690_c368c88b512a7417d739fbfa0dd3278b.jpg temp/1748311828_2794098_1361045626_b7917f136610db026b7e28a884ae996e.jpg temp/1748311828_2794098_1361045618_ac8aaea0068209dee880239962f0ebda.jpg temp/1748311828_2794098_1361045613_9e4f06cf067712859864fa4d491bbd6f.jpg temp/1748311828_2794098_1361045609_43c168381797a05a8e400418608af66f.jpg temp/1748311828_2794098_1361045606_9b6d1a5ca36e0572dc5c816ba069ce92.jpg temp/1748311828_2794098_1361045602_1ec2c47720e3725704c8980798f46436.jpg temp/1748311828_2794098_1361045592_da0568e5f76f5fc8a114cfe20525a676.jpg temp/1748311828_2794098_1361045588_5f709917fe4ecb981ee11bf81fbe2829.jpg temp/1748311828_2794098_1361045585_ac4e8ae13c1c82d4f3d7e180f5181418.jpg temp/1748311828_2794098_1361045581_7c4c4009026ed5ea05397b2b81a01b2b.jpg temp/1748311828_2794098_1361045579_ad0079e0037f398464f8ca8f7f945a6e.jpg temp/1748311828_2794098_1361045576_ddfdad47bf717da1769f16d14c39f4d0.jpg temp/1748311828_2794098_1361045552_6c5c8129926e0fbc76b5537c27309f22.jpg temp/1748311828_2794098_1361045547_48a56208ba34962a3094439903fee869.jpg temp/1748311828_2794098_1361045543_fcc137d47ac6ab4057b10395759cafb1.jpg temp/1748311828_2794098_1361045540_5a61575fb580a95573f9fe49c851b151.jpg temp/1748311828_2794098_1361045536_9a90a5c1f2605f0b99a85aa8cb9cf148.jpg temp/1748311828_2794098_1361045532_16ec8eac32f1c14680ef4e737aba85d4.jpg temp/1748311828_2794098_1361045504_fbd777be73b8e966062fadac47f2357e.jpg temp/1748311828_2794098_1361045500_3f904ae70d3289df3245978a30aabfa2.jpg temp/1748311828_2794098_1361045497_2edf4e6d8d6e25efe792c0ba47b9feb2.jpg temp/1748311828_2794098_1361045496_0d807780781a62ea5c8b7a5cf9235fd7.jpg temp/1748311828_2794098_1361045494_93c35da4d996e7c0aa6e5e12530aa193.jpg temp/1748311828_2794098_1361045491_e504169a8f64767089d846aa72767520.jpg temp/1748311828_2794098_1361045422_2278aa94917428af1d5aca1ac54b0d1c.jpg temp/1748311828_2794098_1361045416_d979dff932efa72d21007e6b2c868c35.jpg temp/1748311828_2794098_1361045413_debc4cefe68b269f21b39688249a448b.jpg temp/1748311828_2794098_1361045408_d6ce2e68aec1ef32225f897f6685161e.jpg temp/1748311828_2794098_1361045404_a2663c0dfa216a187431068debdafae0.jpg temp/1748311828_2794098_1361045400_8c9fea062fce9cdaf0caa8b597a09f94.jpg temp/1748311828_2794098_1361045377_6b631c127fcffc66eebf83fc22db2f9b.jpg temp/1748311828_2794098_1361045373_3b5229d8d6a61bdb7344fc7f161b7662.jpg temp/1748311828_2794098_1361045369_5af711c3dd91ebc939ebf5c3b4f1583d.jpg temp/1748311828_2794098_1361045365_c5c33345a6e138e8b7e08d03475271de.jpg temp/1748311828_2794098_1361045361_d72d1291c564344be580eb0c551eb6e3.jpg temp/1748311828_2794098_1361045357_295a7c49ded2f1ddd91c74176a0e26ba.jpg temp/1748311828_2794098_1361045316_a1330a5eedd3fd651caa193badef3f87.jpg temp/1748311828_2794098_1361045315_ed703bfcf017e183b88d634242ba51f2.jpg temp/1748311828_2794098_1361045313_1d194d10d4c57a85452b2f915c297257.jpg temp/1748311828_2794098_1361045312_cf4dcd65d627eb82d27e168b24646586.jpg temp/1748311828_2794098_1361045310_fc7f162c3e66d21d7029f0fceb557096.jpg temp/1748311828_2794098_1361045308_9c651c19fe010ef93e5a3841f20b9456.jpg temp/1748311828_2794098_1361045301_0ee63edaf568b255b00fb21207134dc5.jpg temp/1748311828_2794098_1361045298_140c010fbe6fc6cb6a19acf0c8293404.jpg temp/1748311828_2794098_1361045294_741c17315294562d9b3d24538bc7b473.jpg temp/1748311828_2794098_1361045292_e477236d0c34e841bf8f3be67157b405.jpg temp/1748311828_2794098_1361045290_3c2998c0182e19987c21056556969f1b.jpg temp/1748311828_2794098_1361045287_051544044d404f5a8c42011fe890b23e.jpg temp/1748311828_2794098_1361045265_51f2df554f8e878f59a30dae8fc1eb4a.jpg temp/1748311828_2794098_1361045261_c87e953bf5271742f435279b29a00ae0.jpg temp/1748311828_2794098_1361045258_9889c094bea66890e04bec33eba05dbe.jpg temp/1748311828_2794098_1361045255_8f1dc84444ec1716ab1c67478e79b67b.jpg temp/1748311828_2794098_1361045252_49e46d17b85c06af2fd2da502e010fab.jpg temp/1748311828_2794098_1361045250_f50cd5ee1a82fed873b6800ccede3c05.jpg temp/1748311828_2794098_1361045229_1fc9f0ca1061a656ca24c7c98ebe5773.jpg temp/1748311828_2794098_1361045226_538eb2105ef2ed56b1da90b3b9ee34b1.jpg temp/1748311828_2794098_1361045221_9eaa5527beab244db9ad02c198a1997c.jpg temp/1748311828_2794098_1361045218_0c39a4bbc36d0b3dae91772036ea44cd.jpg temp/1748311828_2794098_1361045214_1ee69029bf7c9eda048b093a2dcae007.jpg 114it [00:00, 1136.96it/s] 200it [00:00, 1303.06it/s]2025-05-27 04:11:21.888774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 04:11:21.922739: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-27 04:11:21.954014: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 04:11:21.959079: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-27 04:11:21.961524: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748311828_2794098_1361045212_431a69c9f53197c4a6cbced6fd16d33e.jpg temp/1748311828_2794098_1361045194_baa36a67c68b91cc337d93eb0e084f54.jpg temp/1748311828_2794098_1361045189_13b2331bb0515cd3e812c882b139f698.jpg temp/1748311828_2794098_1361045184_dc228f91d0c7fbb842419c967ef5d1b6.jpg temp/1748311828_2794098_1361045181_c2a998686c1a5aff672ccec4e6644553.jpg temp/1748311828_2794098_1361045179_26a1ec1daf0a51e02f639c091b58aff9.jpg temp/1748311828_2794098_1361045177_785c80cc99d6fe03102c7d74bc9cfc1c.jpg temp/1748311828_2794098_1361045161_9497500305033d26721fc1393e5fe73d.jpg temp/1748311828_2794098_1361045155_bf152ff4e73488d880259f0f22770439.jpg temp/1748311828_2794098_1361045152_79a442ace9f452fe5d3f897f72812948.jpg temp/1748311828_2794098_1361045149_6047cc007bdb72daa08667eafd65db36.jpg temp/1748311828_2794098_1361045146_4da28b81e790526146b0dcfc7750848c.jpg temp/1748311828_2794098_1361045144_08633633f08888a758241340202f0de7.jpg temp/1748311828_2794098_1361045125_f3f4cfb4b4dad45584dad658fbbe873d.jpg temp/1748311828_2794098_1361045120_0a99738e34cb0989faaf9d226d61b630.jpg temp/1748311828_2794098_1361045117_4a61a2827f80726a83edd5430f57109a.jpg temp/1748311828_2794098_1361045114_2d2d47cfc6d0174724983512082d4de2.jpg temp/1748311828_2794098_1361045111_f9447ba5eb1d4f18b11598b711818867.jpg temp/1748311828_2794098_1361045107_34a15eaceb1e7a17b4cd18596c6ae92e.jpg temp/1748311828_2794098_1361045073_fb984efbeb6667093acf6fa51a519577.jpg temp/1748311828_2794098_1361045069_1a6dfbcc8651d0c37df58c79c63fd415.jpg temp/1748311828_2794098_1361045065_51b384b01fa056d3f2138f3bac5ed969.jpg temp/1748311828_2794098_1361045063_a832f5572e222bec8df420b4c4dc3158.jpg temp/1748311828_2794098_1361045060_5278b54d5152894fc3a86439d2ff8825.jpg temp/1748311828_2794098_1361045057_04e0a385c98f671e9785746f08ae187a.jpg temp/1748311828_2794098_1361044992_6c34996e642cc030006e401b04fd2cc8.jpg temp/1748311828_2794098_1361044986_5e3b9bf03c7304639ae25d46a9bfc4bf.jpg temp/1748311828_2794098_1361044979_7fdc1f2259df96005e5ae561e2c1a47b.jpg temp/1748311828_2794098_1361044974_5d458aa3c14e1429ec6a3b918c9dd829.jpg temp/1748311828_2794098_1361044969_d972a9f6819bcc39b6b5d49a12ac226f.jpg temp/1748311828_2794098_1361044964_b45bb899194c8b99851df05fc1dc434a.jpg temp/1748311828_2794098_1361044943_263f4db6afd8e5491000b25818740731.jpg temp/1748311828_2794098_1361044940_8999ed2981f078436d35de0e9a7ef2f8.jpg temp/1748311828_2794098_1361044936_1f9c32841c8033e44461ed1b3bd61872.jpg temp/1748311828_2794098_1361044933_b138fb8acf8ce105d1d9361db59c64cb.jpg temp/1748311828_2794098_1361044931_ffd11eca8aacbea429e4e24337c86114.jpg temp/1748311828_2794098_1361044928_140518225d834e412fabaa931b1cf12a.jpg temp/1748311828_2794098_1361044904_c470c4c35b97f05af56aa3d9de8fc975.jpg temp/1748311828_2794098_1361044899_063c8023b42a9d89dc54e59f99addc50.jpg temp/1748311828_2794098_1361044893_e24671d897dac58c416ff2d3635160e0.jpg temp/1748311828_2794098_1361044888_f08f60cb2a67793375014a95dab7b2bb.jpg temp/1748311828_2794098_1361044885_0a1488c7dc7c3783460506564ebc5b1b.jpg temp/1748311828_2794098_1361044881_901dc45d8eda7e2e288e42615ac5ba00.jpg temp/1748311828_2794098_1361044854_dd58bf809e412b7013f0fadc7345aeb0.jpg temp/1748311828_2794098_1361044851_0e1b85c3786b3ee612c35805d4cf0290.jpg temp/1748311828_2794098_1361044847_7bef3fcbd747de7030ec9387ebb1ee5a.jpg temp/1748311828_2794098_1361044844_e894cd22a52bed15b3913b5fac6f0a03.jpg temp/1748311828_2794098_1361044842_d9e0ebd2a723c1ee41a8efa4faf598be.jpg temp/1748311828_2794098_1361044839_ca42e0bf7720f719a6daee9e4a2b306c.jpg temp/1748311828_2794098_1361044818_8d4ed3abf755426593c58c326bd495a2.jpg temp/1748311828_2794098_1361044813_e0b2c6287e73d55a3a235c367445c1f5.jpg temp/1748311828_2794098_1361044808_166750415f27dec5851ba53fa5a33f6e.jpg temp/1748311828_2794098_1361044805_9692474d7655c9288cc5f7ec2b4e66ba.jpg temp/1748311828_2794098_1361044801_ccf27c0f9e78f0fb9d2499f509a71cc7.jpg temp/1748311828_2794098_1361044798_b52ee312ab0499019553515e2f2e9623.jpg temp/1748311828_2794098_1361044778_8874f978eeeda10ee09ce8b5e802cb67.jpg temp/1748311828_2794098_1361044775_19736c484c11287249632a0ad49ec092.jpg temp/1748311828_2794098_1361044771_8069d046a9d6114688c3b183e72ee2b1.jpg temp/1748311828_2794098_1361044769_c89b9758aba05deb7433bed2fb55abf9.jpg temp/1748311828_2794098_1361044767_9bb351346039851f6e2041289a3cab77.jpg temp/1748311828_2794098_1361044764_bc6ba63b430164d817f25489dafa2b9c.jpg temp/1748311828_2794098_1361044714_7a68ba25e1c2c4328c0d23b50718a0e0.jpg temp/1748311828_2794098_1361044712_7a75533a379e199a3538d0889e962f5a.jpg temp/1748311828_2794098_1361044705_c0d4abf582f572e8e281a9d6a03fd38b.jpg temp/1748311828_2794098_1361044701_5699a3c2940416ead20b41604f819531.jpg temp/1748311828_2794098_1361044698_d95b8360209ede128e9a343486516a81.jpg temp/1748311828_2794098_1361044695_ac9c086439f00c5080aabff5b65e4bb1.jpg temp/1748311828_2794098_1361044689_96bb3dd148ca97a7188ce98cc7c7dc30.jpg temp/1748311828_2794098_1361044686_60826c66129c044ecc3b1faf157bbe30.jpg temp/1748311828_2794098_1361044683_9bc7a04e44d6ff059c9ec130b8fe6325.jpg temp/1748311828_2794098_1361044679_0404a85c2a28eddf3e83c9350a6c6ac0.jpg temp/1748311828_2794098_1361044677_96f4ebefc6cf0dabcc158c15d8f59135.jpg temp/1748311828_2794098_1361044675_5c7f378254d486a9b4527de6724252ee.jpg temp/1748311828_2794098_1361044651_9af51f2011c1d367a201f8b4c69569fb.jpg temp/1748311828_2794098_1361044647_993cae072a342dfbf1b6a97eecd57d0f.jpg temp/1748311828_2794098_1361044643_54aeaf81ae13166581174fe7b49a792f.jpg temp/1748311828_2794098_1361044639_d6114d45404a7bc2e072d77ad7962648.jpg temp/1748311828_2794098_1361044635_a50710f4d7ede2127afcc70c31a43184.jpg temp/1748311828_2794098_1361044633_c6ad39d7616da1e2cc4cb98901e1eaa4.jpg temp/1748311828_2794098_1361044605_5f00eff947229487f152a4bc32d32e69.jpg temp/1748311828_2794098_1361044601_e84dde765a2d483fd9e62b724e515438.jpg temp/1748311828_2794098_1361044598_c4478036a1f0250fd3eecad350012748.jpg temp/1748311828_2794098_1361044594_2f4e63976bb8ba62d90a49644d23f81c.jpg temp/1748311828_2794098_1361044592_8fd2e891b1fb7071a416748159d0677e.jpg temp/1748311828_2794098_1361044589_d2f0c181f88955647f729fd4d570a786.jpg temp/1748311828_2794098_1361044551_b84c2be6b935211f9a5b2870f75a16de.jpg temp/1748311828_2794098_1361044546_baa340a56eca90e71f05554505623ae3.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361046093, 1361046088, 1361046086, 1361046084, 1361046082, 1361046080, 1361046039, 1361046035, 1361046031, 1361046026, 1361046021, 1361046018, 1361046008, 1361046006, 1361046005, 1361046004, 1361046003, 1361045999, 1361045938, 1361045934, 1361045929, 1361045927, 1361045910, 1361045909, 1361045871, 1361045869, 1361045868, 1361045867, 1361045866, 1361045865, 1361045842, 1361045838, 1361045836, 1361045832, 1361045829, 1361045826, 1361045780, 1361045778, 1361045775, 1361045772, 1361045768, 1361045765, 1361045756, 1361045752, 1361045749, 1361045746, 1361045743, 1361045740, 1361045715, 1361045710, 1361045705, 1361045701, 1361045696, 1361045690, 1361045626, 1361045618, 1361045613, 1361045609, 1361045606, 1361045602, 1361045592, 1361045588, 1361045585, 1361045581, 1361045579, 1361045576, 1361045552, 1361045547, 1361045543, 1361045540, 1361045536, 1361045532, 1361045504, 1361045500, 1361045497, 1361045496, 1361045494, 1361045491, 1361045422, 1361045416, 1361045413, 1361045408, 1361045404, 1361045400, 1361045377, 1361045373, 1361045369, 1361045365, 1361045361, 1361045357, 1361045316, 1361045315, 1361045313, 1361045312, 1361045310, 1361045308, 1361045301, 1361045298, 1361045294, 1361045292, 1361045290, 1361045287, 1361045265, 1361045261, 1361045258, 1361045255, 1361045252, 1361045250, 1361045229, 1361045226, 1361045221, 1361045218, 1361045214, 1361045212, 1361045194, 1361045189, 1361045184, 1361045181, 1361045179, 1361045177, 1361045161, 1361045155, 1361045152, 1361045149, 1361045146, 1361045144, 1361045125, 1361045120, 1361045117, 1361045114, 1361045111, 1361045107, 1361045073, 1361045069, 1361045065, 1361045063, 1361045060, 1361045057, 1361044992, 1361044986, 1361044979, 1361044974, 1361044969, 1361044964, 1361044943, 1361044940, 1361044936, 1361044933, 1361044931, 1361044928, 1361044904, 1361044899, 1361044893, 1361044888, 1361044885, 1361044881, 1361044854, 1361044851, 1361044847, 1361044844, 1361044842, 1361044839, 1361044818, 1361044813, 1361044808, 1361044805, 1361044801, 1361044798, 1361044778, 1361044775, 1361044771, 1361044769, 1361044767, 1361044764, 1361044714, 1361044712, 1361044705, 1361044701, 1361044698, 1361044695, 1361044689, 1361044686, 1361044683, 1361044679, 1361044677, 1361044675, 1361044651, 1361044647, 1361044643, 1361044639, 1361044635, 1361044633, 1361044605, 1361044601, 1361044598, 1361044594, 1361044592, 1361044589, 1361044551, 1361044546] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.12806129455566406 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 18.40846562385559 time spend to save output : 0.1380913257598877 total time spend for step 0 : 18.54655694961548 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.85user 4.36system 0:57.87elapsed 31%CPU (0avgtext+0avgdata 1724476maxresident)k 32176inputs+272424outputs (113major+551073minor)pagefaults 0swaps