python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 347877 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7542>200 please execute split_portfolio.py -i 23311170 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2946880'] with mtr_portfolio_ids : ['23311170'] and first list_photo_ids : [] new path : /proc/347877/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 33.04744219779968 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon May 26 16:41:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2244 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1445 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1051 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1051 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1051 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1051 wait 20 seconds 2025-05-26 16:43:14.393743: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-26 16:43:14.436898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 16:43:14.438959: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 16:43:14.472152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 16:43:14.493399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 16:43:14.497677: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 16:43:14.531420: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 16:43:14.537415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 16:43:14.596248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 16:43:14.597457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 16:43:14.598470: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-26 16:43:14.631136: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-26 16:43:14.632935: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc4cc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-26 16:43:14.632978: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-26 16:43:14.768780: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x251a1090 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-26 16:43:14.768847: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-26 16:43:14.769989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 16:43:14.770109: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 16:43:14.770145: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 16:43:14.770177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 16:43:14.770208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 16:43:14.770241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 16:43:14.770272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 16:43:14.770304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 16:43:14.771667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 16:43:14.772183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 16:43:14.773354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 16:43:14.773386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 16:43:14.773400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 16:43:14.774528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 1051 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 10.527228355407715 time used to load_weights : 0.6775479316711426 0it [00:00, ?it/s] 58it [00:00, 579.00it/s]temp/1748270430_347877_1360974302_7cd319346e5e6387ed5492769bcd6adf.jpg temp/1748270430_347877_1360974299_1290c63e8799a23aca3caa37f1491945.jpg temp/1748270430_347877_1360974296_415876a42522cbe2f024644218917da5.jpg temp/1748270430_347877_1360974294_d11d293f7b597d68b8cd6eb6a426a1d1.jpg temp/1748270430_347877_1360974293_a597d078801fd2c5c0629defc7b62eda.jpg temp/1748270430_347877_1360974291_f5cbb2a85cdeff165daf512e2109e3b2.jpg temp/1748270430_347877_1360974272_f8ffb0d9eb55dcafbfcf9b7e71d9ce85.jpg temp/1748270430_347877_1360974270_ffe22f8968c2bb2bbd642bebe378bfe8.jpg temp/1748270430_347877_1360974269_e12f6633447baa3204b6d92b635e4c9b.jpg temp/1748270430_347877_1360974268_134a3668e84f7b9282d536411c73e501.jpg temp/1748270430_347877_1360974266_d8ff019e6371fb1052ef961c8efcce65.jpg temp/1748270430_347877_1360974263_6fafa89b07488c6cedb8b84a91c907ea.jpg temp/1748270430_347877_1360974207_8d21cc0d91e9232957a610783204b88c.jpg temp/1748270430_347877_1360974205_88f2b9122cb6ec60cbcec00bfa57d4fc.jpg temp/1748270430_347877_1360974203_ae829fbf43c319af354f05e1eac29e1e.jpg temp/1748270430_347877_1360974201_40568aa311ac71752c4b509e4cf324c6.jpg temp/1748270430_347877_1360974198_c35e23d9e96cce0fbe4ecb980a8ffd33.jpg temp/1748270430_347877_1360974194_9aa853dafc17aef6393334894d5d86b2.jpg temp/1748270430_347877_1360974177_3702ad035c0c550f5615c5003b7d4fe8.jpg temp/1748270430_347877_1360974174_6dbe9c081fb9aa3a1b51e8dab84c7326.jpg temp/1748270430_347877_1360974171_7f0b8db9b3818331d420fa42b1976b09.jpg temp/1748270430_347877_1360974168_44694cc6be74e5e0d113fdb1b5cf13a4.jpg temp/1748270430_347877_1360974166_a9e0147f4e0c5037edaeb76ce4230bf9.jpg temp/1748270430_347877_1360974163_f62d1845ff6e27ad14ffca6d9ad199d9.jpg temp/1748270430_347877_1360974131_531bb2977d47a815bfc606b5334917d5.jpg temp/1748270430_347877_1360974127_bf168b64a966dbaa1ee09c7a46d8a391.jpg temp/1748270430_347877_1360974123_552471c2bab9f0ddbb42d6632350ff4d.jpg temp/1748270430_347877_1360974119_cea38ab1a4ee1d047e6feb6341836330.jpg temp/1748270430_347877_1360974116_699c59de8483e84239096f9dab709114.jpg temp/1748270430_347877_1360974113_28f4291854bc3559cbc545aabf626c83.jpg temp/1748270430_347877_1360974108_b0e0c497ae29a50ab5a11f863f38223d.jpg temp/1748270430_347877_1360974107_e4ebf48ee93c17ae8de24ebfa4d86012.jpg temp/1748270430_347877_1360974106_831f8dbe618d02d514f4900586403e1e.jpg temp/1748270430_347877_1360974105_1a93f73f08b306d9014ae08632ace337.jpg temp/1748270430_347877_1360974103_c3d00c817e7b297d1e2ca52713bed976.jpg temp/1748270430_347877_1360974099_fc11ce7db92aa7c647da2f7cda838f61.jpg temp/1748270430_347877_1360974037_fe01062b7efc5d68681cccf3e3ce9a85.jpg temp/1748270430_347877_1360974033_00cd759a3bfc1ef499d7470f8c8b6dde.jpg temp/1748270430_347877_1360974030_6109d3aab32276294d5fb5f7d19902a5.jpg temp/1748270430_347877_1360974028_85a3ef6883c0be00db2bfaa7e489ccb9.jpg temp/1748270430_347877_1360974025_4e99892a69079a09e123b7398de8dacf.jpg temp/1748270430_347877_1360974023_61e6bcaab07cf6896d09d0a1ba919391.jpg temp/1748270430_347877_1360974006_2cdd50991519b30552cd02d09acdba66.jpg temp/1748270430_347877_1360974004_cd3b2cd49525131555489a0ea5c1bceb.jpg temp/1748270430_347877_1360974002_48286eae807ccdb7d48d4d3b08655e1e.jpg temp/1748270430_347877_1360974000_dd0d22038678e9168b2a9c521a99e1d4.jpg temp/1748270430_347877_1360973998_df882787f084ae6470685e015a64aae2.jpg temp/1748270430_347877_1360973996_92d9e6f3f9a2322186e25330f2f01a63.jpg temp/1748270430_347877_1360973994_f0ddcdb840543e70d17d6771d53f12ae.jpg temp/1748270430_347877_1360973993_0c762b704e3cc95c58818d656763bb14.jpg temp/1748270430_347877_1360973992_7ddb6cb7735ac07c66bbb668f2c25266.jpg temp/1748270430_347877_1360973990_9ed256f9d767aea674d8f561cc440e15.jpg temp/1748270430_347877_1360973989_e4055d93afd072d63b896b6357ef36b7.jpg temp/1748270430_347877_1360973988_1239792ef32d8f084bd25d62b0fe7534.jpg temp/1748270430_347877_1360973970_96dc1df5cd7d5cb37f1465024e3201e3.jpg temp/1748270430_347877_1360973968_7ebab23ca07bb9281741effc8c0df38d.jpg temp/1748270430_347877_1360973967_ccaf414c1cda8bf0566fc83759ac6b94.jpg temp/1748270430_347877_1360973966_516c4734e0f424df20bbe187e6bfc62b.jpg temp/1748270430_347877_1360973963_cf4668524176420cd71a6e4dd25dcc23.jpg temp/1748270430_347877_1360973961_ceeecb1f46c89e1c3acc044582fc45a6.jpg temp/1748270430_347877_1360973944_b5681d288e81cc8c890b7cebd2062eaa.jpg temp/1748270430_347877_1360973940_0e1fe697c8444b7f78520aa8d799bbf5.jpg temp/1748270430_347877_1360973937_6909e111a3f48d4f097871e7534b5746.jpg temp/1748270430_347877_1360973934_45b14058001c2f0a9415c8ae2d7dcce9.jpg temp/1748270430_347877_1360973931_4d79b31c753ed8add4e35c5e6f1bb72e.jpg temp/1748270430_347877_1360973927_cb3cf760e0df1a49660a3613fd71974b.jpg temp/1748270430_347877_1360973911_20a94a4caaaae352dbd31aaa90732483.jpg temp/1748270430_347877_1360973909_b8806fb74ad770334766a53817edc309.jpg temp/1748270430_347877_1360973907_d7ea17da27c51a7b1fd845b61d6ee6a7.jpg temp/1748270430_347877_1360973906_95439e14c4717aafccc77f93a6a7926b.jpg temp/1748270430_347877_1360973905_7ef53747fd93a407692bcd432c178599.jpg temp/1748270430_347877_1360973903_fac66a772314e0160462cafde3b7e6a0.jpg temp/1748270430_347877_1360973885_6c931e085de3895606aa648c3bc85880.jpg temp/1748270430_347877_1360973882_568f8a1560eb2aa35e794923238b1361.jpg temp/1748270430_347877_1360973880_17ff053af8eff0f926d0ac6d3fe187a4.jpg temp/1748270430_347877_1360973878_7960d6963759b8918beb5c7f553af43d.jpg temp/1748270430_347877_1360973876_790650809364e2157ebf3d6709f42186.jpg temp/1748270430_347877_1360973873_4ed7f5b138a3bf36c8035eb1865f0d26.jpg temp/1748270430_347877_1360973851_4089d659e6a4a15ddb8ea7f0eba0ae37.jpg temp/1748270430_347877_1360973849_f1bfadd0390055dce4d93c0bea5a76a5.jpg temp/1748270430_347877_1360973848_93693633c38411f661d6aa60118b3644.jpg temp/1748270430_347877_1360973847_4f45b4ba2cb16560c308d2f62fb5bd3e.jpg temp/1748270430_347877_1360973846_e1f6059ace7d4940ba4c2dfacefbbe05.jpg temp/1748270430_347877_1360973842_96aa846d50f29d104918e29d5857c50c.jpg temp/1748270430_347877_1360973821_2cb2291ff8d3a829cad236d058da7a44.jpg temp/1748270430_347877_1360973818_e3a91e630cdf03f61bb184dbafa9cd11.jpg temp/1748270430_347877_1360973816_931e0aaea4ea7c34e8da855ae6f82c0e.jpg temp/1748270430_347877_1360973814_25415b7bff381b5c5b8d94392104aa5b.jpg temp/1748270430_347877_1360973811_c54c0bceae708ce01a45d95b0112c093.jpg temp/1748270430_347877_1360973807_122e357e66bb69a6fbae724af657cec8.jpg temp/1748270430_347877_1360973791_b6c3b2c40bd788dc389488d1cae220c1.jpg temp/1748270430_347877_1360973788_cd9a08e9ae57625e19cce9ca75fcac90.jpg temp/1748270430_347877_1360973786_e2396e53358c39aa8af8dfdeb38bbeba.jpg temp/1748270430_347877_1360973784_179de8a77d5dc4b942a4a4d21a23cdc6.jpg temp/1748270430_347877_1360973782_6836cf36b62763b1c5d27e0344978c45.jpg temp/1748270430_347877_1360973780_41925952f003db245c175c15d8fb404c.jpg temp/1748270430_347877_1360973762_1c1b0fdfc1b5dd721ed9bb60740afd02.jpg temp/1748270430_347877_1360973760_cee511322bd657e485669f8a362d309c.jpg temp/1748270430_347877_1360973759_b0a7b2f4a50a4021712acc3eb4d2b944.jpg temp/1748270430_347877_1360973758_f5592eb419f7e8f51fd8ed5e702a9eba.jpg temp/1748270430_347877_1360973756_fb4842d847fa8d8e444cd0b2d20a27af.jpg temp/1748270430_347877_1360973753_255f48d0238892b14732614a2db11133.jpg temp/1748270430_347877_1360973735_7601f769d8c9970b0fdd4480740044a2.jpg temp/1748270430_347877_1360973732_9563c175b0e6caa118991009d242ec5d.jpg temp/1748270430_347877_1360973729_33c2b3f9c36c0d2e73082fe3cb8edada.jpg temp/1748270430_347877_1360973726_07a6fbeddc7b7723f446c5d55ca50821.jpg temp/1748270430_347877_1360973723_ea1b58397c0656e3bba33dbce79f6be0.jpg temp/1748270430_347877_1360973718_108b8ace48db0c3dceccbaad2a3a96f4.jpg temp/1748270430_347877_1360973703_3369dd30c2f1efec39b62636f4499d3d.jpg temp/1748270430_347877_1360973700_69d6342d3d218ce63190e557cc3b3410.jpg temp/1748270430_347877_1360973698_ecfebd33462a64184f4ae37044c3b86f.jpg temp/1748270430_347877_1360973697_3ffd778f4fb39fb9f7282f800463294c.jpg temp/1748270430_347877_1360973694_8b4e7ef1ae2567a30a4c74fb434214db.jpg temp/1748270430_347877_1360973692_1f659e87d88d271017596267f1d4f678.jpg temp/1748270430_347877_1360973675_681172e06b7e3b216b12142f6cf2ad84.jpg 124it [00:00, 623.42it/s] 192it [00:00, 646.49it/s] 200it [00:00, 638.15it/s]2025-05-26 16:43:28.778201: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 16:43:29.100208: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-26 16:43:29.130035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 16:43:29.141381: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-26 16:43:29.145480: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748270430_347877_1360973673_033aa4ef90481e6f018a953c066b314c.jpg temp/1748270430_347877_1360973671_79e7bf41f19a3c520f7899402271487d.jpg temp/1748270430_347877_1360973669_ecc7731984185acdbac3173a6a4a2f08.jpg temp/1748270430_347877_1360973666_486dbbab5c58c763c3e0d8da007dfb90.jpg temp/1748270430_347877_1360973662_f009f588a68f3187539bf0140ed97c98.jpg temp/1748270430_347877_1360973644_23c2e619d243263461174931d15cc039.jpg temp/1748270430_347877_1360973641_003a3900412510d03a5fb8c127c5a584.jpg temp/1748270430_347877_1360973639_060b071d14efe84005e7511703bb394c.jpg temp/1748270430_347877_1360973638_4ebad2a4783ae6d3949db216410c26c1.jpg temp/1748270430_347877_1360973637_6ee46c04bc121711d5f9e29024252859.jpg temp/1748270430_347877_1360973634_6973817afd604e4b67b4729b50295fb4.jpg temp/1748270430_347877_1360973616_bd89879967e890c99cc6f03a736ea7c7.jpg temp/1748270430_347877_1360973614_0417fe280067b770091547c3b38e944b.jpg temp/1748270430_347877_1360973612_8393b68a50252a03784fdbd0b1eab3bd.jpg temp/1748270430_347877_1360973610_bffb1096471a68158ba8488a99337ac7.jpg temp/1748270430_347877_1360973609_15c4882d511c2bea3d1c5269b33963fb.jpg temp/1748270430_347877_1360973606_675b58c346d8628f7e323d7eb7c979e9.jpg temp/1748270430_347877_1360973587_20206a35150a03b82a9e3ea717055ac9.jpg temp/1748270430_347877_1360973584_deb0dd34f84ee84fe2d250752f765f71.jpg temp/1748270430_347877_1360973582_8673dc83dfd2247f8994a2a20fda2843.jpg temp/1748270430_347877_1360973580_9972b61eaaecb6310f1f95d803e8b74c.jpg temp/1748270430_347877_1360973578_e1284087aa6b14952127bfe2da008d41.jpg temp/1748270430_347877_1360973575_86195cb691874484faa979b9f1067edf.jpg temp/1748270430_347877_1360973534_9f97a0fc4dd36c0415dc3c25e85e635d.jpg temp/1748270430_347877_1360973530_824e918e8702c99783171e05bfbdaf25.jpg temp/1748270430_347877_1360973527_db332550295f00fe40b8aaa01f366050.jpg temp/1748270430_347877_1360973525_5120a699184c79b1505f64854e256495.jpg temp/1748270430_347877_1360973524_681ccf07f39ef4b6eb9ce319df740899.jpg temp/1748270430_347877_1360973523_df0a39fffaafe3eec45509c7ad911fcc.jpg temp/1748270430_347877_1360973517_2970a3c4f3ed64bc3d4acb5ceb0c4bfc.jpg temp/1748270430_347877_1360973514_90b1d547e57b78d83278f73859ac0def.jpg temp/1748270430_347877_1360973511_4b32306071fcdbbc0ec8c7d872d1d7d9.jpg temp/1748270430_347877_1360973509_d0db20bf4d545932dea9c2cb0b8495e2.jpg temp/1748270430_347877_1360973507_d3aee6c767562d2bc0f1d0df2da40342.jpg temp/1748270430_347877_1360973504_fa18a3dc7e4e0e5423e5878278b1199e.jpg temp/1748270430_347877_1360973469_8bb733acf9aa5edd546c48fa5c14f33f.jpg temp/1748270430_347877_1360973466_5b7cb4c9b94afbd48fdb505ae61bfb44.jpg temp/1748270430_347877_1360973462_a123d6e9a065203dd8e4cc635eef99f6.jpg temp/1748270430_347877_1360973458_9f74a97417b1d7f1def58a1d24331619.jpg temp/1748270430_347877_1360973455_cdd07687d96211136c46b63766dce816.jpg temp/1748270430_347877_1360973452_2ec300ffdafd75882ab452d799961429.jpg temp/1748270430_347877_1360973448_126fb3761c37adf8a46bd326ddbd77e4.jpg temp/1748270430_347877_1360973447_e17ef0f460cd273c58beda11eee9954e.jpg temp/1748270430_347877_1360973446_253bb14b70a804294b53898017afaecf.jpg temp/1748270430_347877_1360973445_0ca079c5c00945218a39e0dfdb53a296.jpg temp/1748270430_347877_1360973442_a84b893c6678779ea1ca74cbb27e3391.jpg temp/1748270430_347877_1360973438_cc3d9841e2704345e5ea2745f16ebea7.jpg temp/1748270430_347877_1360973419_54d143698eb7f56dd0733cd7602de692.jpg temp/1748270430_347877_1360973416_0b49523b6ac921fd3a84364bbc127870.jpg temp/1748270430_347877_1360973413_001f816e56c0535b49442e4f5f57007d.jpg temp/1748270430_347877_1360973410_4aef58712b859ab1ff582c288b0979ea.jpg temp/1748270430_347877_1360973407_e982197a88bf1c1714e8645b06f7105c.jpg temp/1748270430_347877_1360973405_67f40438fffa056bd7278fec393aa130.jpg temp/1748270430_347877_1360973388_eb5a611aac3d51a89948118ab23b404d.jpg temp/1748270430_347877_1360973385_73e51b6f03311240c840d41a90adba7c.jpg temp/1748270430_347877_1360973383_7dc1de32b5bacfd4e45b3bec7a33c5c9.jpg temp/1748270430_347877_1360973382_486076181993e532825508a56ae332cd.jpg temp/1748270430_347877_1360973381_0d1e358f843ca7cb1cc12b370855a48a.jpg temp/1748270430_347877_1360973378_2defdf3307127740e7f620f87c61aed6.jpg temp/1748270430_347877_1360973341_5422db707435cbd8e3b7324a023cd52b.jpg temp/1748270430_347877_1360973339_ec92e8c1b8b8b0c11ed0b2b288064c37.jpg temp/1748270430_347877_1360973336_f7eb603c159c004cdd76a88db129a07c.jpg temp/1748270430_347877_1360973332_804ca4a6086203ad88d2999b0141c5a4.jpg temp/1748270430_347877_1360973329_324797e665399407ebfbb9343ce70eef.jpg temp/1748270430_347877_1360973326_09bb32e05a7b0fe99aa62d20884e6ec0.jpg temp/1748270430_347877_1360973322_13c6d18a078bfe1cef91f1d7031d2f75.jpg temp/1748270430_347877_1360973320_351a15833955bf381f6d9834d31968eb.jpg temp/1748270430_347877_1360973319_ad6ba0951eb0d7b1f358f36408d30dbf.jpg temp/1748270430_347877_1360973318_2e6fa30d3ccc8d7654c54316e0fbb776.jpg temp/1748270430_347877_1360973316_e0b2c1e2c2055dc86882b213a975c7db.jpg temp/1748270430_347877_1360973313_999d3cf9301ba52c4a2449da4002c011.jpg temp/1748270430_347877_1360973295_7725b7757111d490990385c8125eba81.jpg temp/1748270430_347877_1360973292_c9907942786a3187bf8a977a9de93fb6.jpg temp/1748270430_347877_1360973289_25fae07d5cdbca29398c1b649eebc362.jpg temp/1748270430_347877_1360973286_813f65f7ba63d988abc12762d6bf3b6f.jpg temp/1748270430_347877_1360973283_4fdd967605d2fc69901caf236b32307c.jpg temp/1748270430_347877_1360973280_502afc146017405f1fa644a987a19967.jpg temp/1748270430_347877_1360973263_c7b73595e25fa4a83742f433b36a1c7a.jpg temp/1748270430_347877_1360973261_858ac46554a3cf508d20bc44be3361b4.jpg temp/1748270430_347877_1360973260_e1e3d92b3bb0e05e1e7ab8a4b82941f4.jpg temp/1748270430_347877_1360973259_472392601e671252909cdd2b8911c885.jpg temp/1748270430_347877_1360973257_653f75299fd2ef83d3e880c816ccb61b.jpg temp/1748270430_347877_1360973255_d09f849b71ecd0354bc9bb8f87723883.jpg temp/1748270430_347877_1360973234_a085dfec4d9cebd5e557f919301931e0.jpg temp/1748270430_347877_1360973231_47b9814b510214330cbb87e728770382.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1360974302, 1360974299, 1360974296, 1360974294, 1360974293, 1360974291, 1360974272, 1360974270, 1360974269, 1360974268, 1360974266, 1360974263, 1360974207, 1360974205, 1360974203, 1360974201, 1360974198, 1360974194, 1360974177, 1360974174, 1360974171, 1360974168, 1360974166, 1360974163, 1360974131, 1360974127, 1360974123, 1360974119, 1360974116, 1360974113, 1360974108, 1360974107, 1360974106, 1360974105, 1360974103, 1360974099, 1360974037, 1360974033, 1360974030, 1360974028, 1360974025, 1360974023, 1360974006, 1360974004, 1360974002, 1360974000, 1360973998, 1360973996, 1360973994, 1360973993, 1360973992, 1360973990, 1360973989, 1360973988, 1360973970, 1360973968, 1360973967, 1360973966, 1360973963, 1360973961, 1360973944, 1360973940, 1360973937, 1360973934, 1360973931, 1360973927, 1360973911, 1360973909, 1360973907, 1360973906, 1360973905, 1360973903, 1360973885, 1360973882, 1360973880, 1360973878, 1360973876, 1360973873, 1360973851, 1360973849, 1360973848, 1360973847, 1360973846, 1360973842, 1360973821, 1360973818, 1360973816, 1360973814, 1360973811, 1360973807, 1360973791, 1360973788, 1360973786, 1360973784, 1360973782, 1360973780, 1360973762, 1360973760, 1360973759, 1360973758, 1360973756, 1360973753, 1360973735, 1360973732, 1360973729, 1360973726, 1360973723, 1360973718, 1360973703, 1360973700, 1360973698, 1360973697, 1360973694, 1360973692, 1360973675, 1360973673, 1360973671, 1360973669, 1360973666, 1360973662, 1360973644, 1360973641, 1360973639, 1360973638, 1360973637, 1360973634, 1360973616, 1360973614, 1360973612, 1360973610, 1360973609, 1360973606, 1360973587, 1360973584, 1360973582, 1360973580, 1360973578, 1360973575, 1360973534, 1360973530, 1360973527, 1360973525, 1360973524, 1360973523, 1360973517, 1360973514, 1360973511, 1360973509, 1360973507, 1360973504, 1360973469, 1360973466, 1360973462, 1360973458, 1360973455, 1360973452, 1360973448, 1360973447, 1360973446, 1360973445, 1360973442, 1360973438, 1360973419, 1360973416, 1360973413, 1360973410, 1360973407, 1360973405, 1360973388, 1360973385, 1360973383, 1360973382, 1360973381, 1360973378, 1360973341, 1360973339, 1360973336, 1360973332, 1360973329, 1360973326, 1360973322, 1360973320, 1360973319, 1360973318, 1360973316, 1360973313, 1360973295, 1360973292, 1360973289, 1360973286, 1360973283, 1360973280, 1360973263, 1360973261, 1360973260, 1360973259, 1360973257, 1360973255, 1360973234, 1360973231] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.099334716796875 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 146.16584944725037 time spend to save output : 0.10758185386657715 total time spend for step 0 : 146.27343130111694 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 16.03user 5.62system 3:05.15elapsed 11%CPU (0avgtext+0avgdata 1833828maxresident)k 2140024inputs+331864outputs (15713major+647332minor)pagefaults 0swaps