python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2998237 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 1876>200 please execute split_portfolio.py -i 27896762 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3957315'] with mtr_portfolio_ids : ['27896762'] and first list_photo_ids : [] new path : /proc/2998237/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 35.2852418422699 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat Oct 18 13:21:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-10-18 13:21:56.388897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-10-18 13:21:56.419230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-18 13:21:56.419466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 13:21:56.421063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 13:21:56.422555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-18 13:21:56.422822: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-18 13:21:56.424524: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-18 13:21:56.425388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-18 13:21:56.428892: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 13:21:56.429884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-18 13:21:56.430223: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-10-18 13:21:56.454499: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-10-18 13:21:56.456054: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa674000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-10-18 13:21:56.456096: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-10-18 13:21:56.556602: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x24233a40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-10-18 13:21:56.556640: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-10-18 13:21:56.557626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-18 13:21:56.557700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 13:21:56.557729: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 13:21:56.557755: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-18 13:21:56.557782: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-18 13:21:56.557808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-18 13:21:56.557834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-18 13:21:56.557860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 13:21:56.559236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-18 13:21:56.559309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 13:21:56.560123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-10-18 13:21:56.560146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-10-18 13:21:56.560159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-10-18 13:21:56.561545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3441 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.347658157348633 time used to load_weights : 0.1851205825805664 0it [00:00, ?it/s] 75it [00:00, 740.24it/s]temp/1760786478_2998237_1389947046_b458da1f454adf94dd96aa9da37393b3.jpg temp/1760786478_2998237_1389946949_545d9e42bc88a3e31e6366bfdf9be5a2.jpg temp/1760786478_2998237_1389946948_77e2ed8fec6e1861541868f6ca1bdf6a.jpg temp/1760786478_2998237_1389946890_711d5cab29a400d3d214c4866bb60776.jpg temp/1760786478_2998237_1389946889_12e5d24e298477af5356fda801af84fe.jpg temp/1760786478_2998237_1389946858_79fb897ef9157f0cbc9bb0343d396946.jpg temp/1760786478_2998237_1389946797_fb12878dbd41a222eb985e8590cd0e17.jpg temp/1760786478_2998237_1389946782_2a146091e62e8967f7f8136b9867b3ab.jpg temp/1760786478_2998237_1389946758_ffc0c5f31b81db94ec2975591c38d5d0.jpg temp/1760786478_2998237_1389946729_233bf49a271cb83186bbee58ac3592f4.jpg temp/1760786478_2998237_1389946707_d53d8ced37673485126b38b7ff86d527.jpg temp/1760786478_2998237_1389946700_fe03d71289e6333cc7e85a2d98650220.jpg temp/1760786478_2998237_1389946676_faf3a28361cc175da6b95d92517e74ed.jpg temp/1760786478_2998237_1389946654_51f72918d37322a115203478a3cef8e5.jpg temp/1760786478_2998237_1389946645_417f33afc1f17ef10e1c6e4dc90bc2f1.jpg temp/1760786478_2998237_1389946634_2846acd806eda7ab613ea30ad48ac051.jpg temp/1760786478_2998237_1389946614_3ced4c04c06bd6a62df080af759b8644.jpg temp/1760786478_2998237_1389946606_4fcffb2a1c2a6c82d158ae4dfa252b3d.jpg temp/1760786478_2998237_1389946593_7b1eb4889a202fe373baa5e52f092b56.jpg temp/1760786478_2998237_1389946554_11918c68abdc2755f83c116a5e40faae.jpg temp/1760786478_2998237_1389946549_f45cc8cea4ee8eb87929457aaba2dad8.jpg temp/1760786478_2998237_1389946534_841a7f375823147db40dda3aa02c7617.jpg temp/1760786478_2998237_1389946531_7d33756540a3d6d9277c6ff137c19b8a.jpg temp/1760786478_2998237_1389946529_abca6d786c35ca307a8ac3042f820bf8.jpg temp/1760786478_2998237_1389946509_33beae4f20a7ed11d0d67bc88b5d98ff.jpg temp/1760786478_2998237_1389946507_8c68b2373f19f4d5ecce335f63c19aba.jpg temp/1760786478_2998237_1389946506_80120698888a5eacbe6a9f83bf248d28.jpg temp/1760786478_2998237_1389946488_4e39310cf5ad481c22517be1cea34a26.jpg temp/1760786478_2998237_1389946432_e683157ced0beff041a8425050bc2cc2.jpg temp/1760786478_2998237_1389946422_0274fbf2240675ebf51e2b6c7aef6232.jpg temp/1760786478_2998237_1389946411_3edd2e7180448eff8474788e20e411d5.jpg temp/1760786478_2998237_1389946403_0dc2e23a3b4ea69b36b0500d2eb79594.jpg temp/1760786478_2998237_1389946399_2990dcac52ce4a37c599163b0d510938.jpg temp/1760786478_2998237_1389946392_fcb7d1bb0630bc9a079fd91a3dbb6e5a.jpg temp/1760786478_2998237_1389946376_d8de6ec5c416cd405d0ea5b957bc40de.jpg temp/1760786478_2998237_1389946307_a429a9c1799d4ac8a7084ae5c0e98f42.jpg temp/1760786478_2998237_1389946263_25c8e6f67b607927a37bb36a4d36ecf1.jpg temp/1760786478_2998237_1389946207_4ea64ade40071dc66c3949b04e4cc622.jpg temp/1760786478_2998237_1389946177_6baa72472e509449dd54189ebe97b7e2.jpg temp/1760786478_2998237_1389946100_441f17e3773a9e55ef1579c74c014ad4.jpg temp/1760786478_2998237_1389946099_9f89ea9b9d733f1fcb096e55325f781d.jpg temp/1760786478_2998237_1389946071_80bbf9111cc07f024d483895cf565903.jpg temp/1760786478_2998237_1389946070_9cddfd850e8be86f125b022f07295520.jpg temp/1760786478_2998237_1389945998_1199b365ceed5a7cb1c47bdcb52a4e8a.jpg temp/1760786478_2998237_1389945997_78855365ebc3db7bb5896a2e3f163726.jpg temp/1760786478_2998237_1389945973_e12ea0b8f7b3cbcd5334583193b8a151.jpg temp/1760786478_2998237_1389945956_cb59aae8b1e33ec6f571b8ebf810a141.jpg temp/1760786478_2998237_1389945896_2179fe3f8219dbb5298fc88b0167101c.jpg temp/1760786478_2998237_1389945895_662c037760d171232b672aef2a0f51eb.jpg temp/1760786478_2998237_1389945894_6060fa4ffcfed211878bc1060b3868e5.jpg temp/1760786478_2998237_1389945893_b69b94e2d5e61604de0ec1637355c63e.jpg temp/1760786478_2998237_1389945892_4928c1672be0445fd5cf228d811f0e44.jpg temp/1760786478_2998237_1389945891_e140a127f561cd0b680d5f6772165107.jpg temp/1760786478_2998237_1389945867_dc1959160fd858a31a59e994ae6593c9.jpg temp/1760786478_2998237_1389945846_44f3d92378d7c22949e1a527ab3a2a94.jpg temp/1760786478_2998237_1389945797_c736b9e843f88a82ced7cfe83e9a3189.jpg temp/1760786478_2998237_1389945796_95bf5b3fd98be9ff633801c8d66cb2ab.jpg temp/1760786478_2998237_1389945795_a326e0dc3e1854f8bf08ebc084a118f4.jpg temp/1760786478_2998237_1389945792_a48c92e7a84f7847188a5c90bcc76aad.jpg temp/1760786478_2998237_1389945791_9e027abff39623d5a860b1d187e34cdf.jpg temp/1760786478_2998237_1389945763_87ae012f23c8b9ebbc70907d22cb126f.jpg temp/1760786478_2998237_1389945761_2ddb1d61e88f8ae506099822d00a0cec.jpg temp/1760786478_2998237_1389945674_7dcdd92ee117628f1f699ff3501c3016.jpg temp/1760786478_2998237_1389945673_dddd016fed6042d55f75c99a8ba21267.jpg temp/1760786478_2998237_1389945672_b26cc551a6653616e05aee7e2e643af5.jpg temp/1760786478_2998237_1389945671_bff89236e55725ece32e885c3b4e3d09.jpg temp/1760786478_2998237_1389945667_eb362f5dbf8fc57134c1916cb1fe3f2f.jpg temp/1760786478_2998237_1389945666_3cdbeb9bb3241a4c8a1b1607421e5b92.jpg temp/1760786478_2998237_1389945661_6421fa26791ada685ee8f468bfdc0de7.jpg temp/1760786478_2998237_1389945656_74f7d5a844d8b84cc27020ca04be1988.jpg temp/1760786478_2998237_1389945643_bff8f8fcc2bfbceb067f124e7fbdbac6.jpg temp/1760786478_2998237_1389945607_df54c6e602f52ccfc8b94a639d607c6b.jpg temp/1760786478_2998237_1389945597_a12eb3794ccd488c10ff921be96697c1.jpg temp/1760786478_2998237_1389945558_59e6b733773ad75f86cb7b3eeb2b31f7.jpg temp/1760786478_2998237_1389945512_df9430656c40d37402b498168d30651e.jpg temp/1760786478_2998237_1389945485_d18dc41b9e89fdf837363c4f77ab502e.jpg temp/1760786478_2998237_1389945481_6d20848d8058280314d4d3b5cc643109.jpg temp/1760786478_2998237_1389945476_26630b1edd9a3f09502648c74106750b.jpg temp/1760786478_2998237_1389945446_c2023e63f573796dead119321454eb8f.jpg temp/1760786478_2998237_1389945393_3e682f404e93692793d5eaf4ced876a7.jpg temp/1760786478_2998237_1389945350_3d42226f868acb996b6c6ddce8d9eedf.jpg temp/1760786478_2998237_1389945291_7acba6da756282a59a9c743eeb666d79.jpg temp/1760786478_2998237_1389945259_58de99f542229e604fea692610d08b05.jpg temp/1760786478_2998237_1389945204_656dd19eca190beefbe2d9a39bd14f78.jpg temp/1760786478_2998237_1389945110_5a3e33c0ba1c659a2fe8378a8358d746.jpg temp/1760786478_2998237_1389945065_1864b3b0656d14984600ece760650f01.jpg temp/1760786478_2998237_1389944983_6da0b50143f45e3d4ee50fe6d37c4696.jpg temp/1760786478_2998237_1389944917_386df8152eb83183dc5661d8da64f19b.jpg temp/1760786478_2998237_1389944916_f476518545f39675a486b3317998a69f.jpg temp/1760786478_2998237_1389944892_1c7a57667fe754b70e315dd7b168406d.jpg temp/1760786478_2998237_1389944829_07d2b3c1ad935979f096ab0b1508e60c.jpg temp/1760786478_2998237_1389944809_62d13bc0892d880ccf99184a74b2ea5a.jpg temp/1760786478_2998237_1389944741_351dc13d3f5d80643789c867038d2534.jpg temp/1760786478_2998237_1389944658_234d53a6339d1ab58c7817acc3b03540.jpg temp/1760786478_2998237_1389944576_eca5dc1a6c0d6cd25c21f4cddd8ae442.jpg temp/1760786478_2998237_1389944462_64822ddfd026124241bd5a88872ed9fa.jpg temp/1760786478_2998237_1389944377_4ec1f46c1731d96ff00010a89f4e1dab.jpg temp/1760786478_2998237_1389944287_b24eb7dfa3ae9e02e2bbc3d6c8fee6f6.jpg temp/1760786478_2998237_1389944275_80660fccca36ed72e88ec777c6fbf6be.jpg temp/1760786478_2998237_1389944206_441152b86bdc75e099d083cfd45e302b.jpg temp/1760786478_2998237_1389944203_9b1535cb8cec2b9b2c715ab2e9c41793.jpg temp/1760786478_2998237_1389944185_90f339e78c41bdcf99efbb52615d159e.jpg temp/1760786478_2998237_1389944123_2ae7ed3b082e356dc1acaa35481a75c7.jpg temp/1760786478_2998237_1389944121_9a03abf4aa4e7387ac99322ef3d9c4b1.jpg temp/1760786478_2998237_1389944097_557e21b2d6fa38262cd47cb7823fdfca.jpg temp/1760786478_2998237_1389944041_c98fd161ae3496fc7e167c83cdcab4da.jpg temp/1760786478_2998237_1389944028_81292d35ed567028d985b936b1b37223.jpg temp/1760786478_2998237_1389944022_21094fd06f7ad1e114afdc3214fb1504.jpg temp/1760786478_2998237_1389943957_6bb6658ae354b71957d0f3fa63162c83.jpg temp/1760786478_2998237_1389943929_e310f77c0b3339c3a5a07395bb3efd0e.jpg temp/1760786478_2998237_1389943874_7e65170054aa0f4a42f0bea19ea8b1f8.jpg temp/1760786478_2998237_1389943839_04d2528e89bc202f94ee0ab281b5e165.jpg temp/1760786478_2998237_1389943837_dcbe126cb43d8f2541dac53f7c9fa627.jpg 163it [00:00, 818.47it/s] 200it [00:00, 826.69it/s]2025-10-18 13:22:06.383931: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 13:22:06.522486: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-10-18 13:22:06.603697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 13:22:06.613884: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-10-18 13:22:06.617252: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1760786478_2998237_1389943788_9b9c9a7cd82404ab99894bfeeae078f4.jpg temp/1760786478_2998237_1389943772_9ebabc87531fa7f9e72c32af6d16742e.jpg temp/1760786478_2998237_1389943754_795d5cdf43ff1e26c4a9351e312e79d2.jpg temp/1760786478_2998237_1389943746_02fe9110ec9ce39e8219b22d5d8c633a.jpg temp/1760786478_2998237_1389943701_f956dd749b700201ce23a4f80c142372.jpg temp/1760786478_2998237_1389943667_f2d55ae3191181298487fe87b60a4435.jpg temp/1760786478_2998237_1389943581_7e4a8406fd9740abd7669f513489062b.jpg temp/1760786478_2998237_1389943554_a6e3242489169dfc9766dbea354484d3.jpg temp/1760786478_2998237_1389943538_46e1f5c26aed5f982fcb5699c788bf8b.jpg temp/1760786478_2998237_1389943495_2af42f632c8055363af511bb3328c6d0.jpg temp/1760786478_2998237_1389943407_e220c8943b5c5fd9e5957f9bec6624d2.jpg temp/1760786478_2998237_1389943268_8e6b38714d313864f424f9dab79e67dd.jpg temp/1760786478_2998237_1389943170_5ae787eb2162008e6825a7ba2ba4f865.jpg temp/1760786478_2998237_1389943077_717f439e2443a240125689a71709b452.jpg temp/1760786478_2998237_1389942992_2f2a9b707f0d89d3516661db539e8b98.jpg temp/1760786478_2998237_1389942897_17180ad08cb5e65976ef236a294690ca.jpg temp/1760786478_2998237_1389942896_88bd91d636f3e98c440d557b20222c34.jpg temp/1760786478_2998237_1389942809_37ba1a41d7535c6e126fc3646227d5a5.jpg temp/1760786478_2998237_1389942804_348ca373bbe03d512d3a34309c3bbcd8.jpg temp/1760786478_2998237_1389942772_04b503ba3588fef468a38d9f892bb028.jpg temp/1760786478_2998237_1389942732_48d0fa529a720b45b6d1ae236c114d08.jpg temp/1760786478_2998237_1389942690_5c78076c1da7e2c5c9df70826d76a262.jpg temp/1760786478_2998237_1389942663_7b41c63bc6e504888ae9f07f917460d1.jpg temp/1760786478_2998237_1389942633_2d918f79f2e8cf185c13d30f008202e9.jpg temp/1760786478_2998237_1389942632_77cd2b127baaef10055a3101b88e58f6.jpg temp/1760786478_2998237_1389942618_65573da0c2bbb95016c8b492bc86917b.jpg temp/1760786478_2998237_1389942583_7bd95e864bc6c24532333440dc45da67.jpg temp/1760786478_2998237_1389942567_bdab4d31a50839a4e8ab2afd8c57daa8.jpg temp/1760786478_2998237_1389942524_d59cb7b413034581cc06525ff9c03b38.jpg temp/1760786478_2998237_1389942510_d7377833fd802f83c9b9d2ce1d790a08.jpg temp/1760786478_2998237_1389942475_2600d43181ae92379e3039364aaff840.jpg temp/1760786478_2998237_1389942463_ba3d369af8e27972ccda2d74fa27336a.jpg temp/1760786478_2998237_1389942438_84b3b5a3615d50388250d6bbbd2b6494.jpg temp/1760786478_2998237_1389942430_789b50a80f46d262e6ae255452b7189e.jpg temp/1760786478_2998237_1389942362_77bbedef8b5fca7e068e37c2501534e1.jpg temp/1760786478_2998237_1389942312_3e2f6d47fd203210b981922c43b52437.jpg temp/1760786478_2998237_1389942259_939b5cfe5497a91b988838e617bec385.jpg temp/1760786478_2998237_1389942019_e6f22f3fb594566a34c34defb76a41ac.jpg temp/1760786478_2998237_1389941919_616fc012b8b1a04f36565a3a347242a2.jpg temp/1760786478_2998237_1389941813_ef122b1095c3316bc16845bcdf19efd4.jpg temp/1760786478_2998237_1389941769_ca949d7d29cd08f9fb5f8fca37336129.jpg temp/1760786478_2998237_1389941700_537b968e360054cd1185496cbe97db9a.jpg temp/1760786478_2998237_1389941652_8a102d036628bd03b38d111371485801.jpg temp/1760786478_2998237_1389941617_aa16ec716906e1eb5aac1d9e5dba8b2d.jpg temp/1760786478_2998237_1389941604_c98f412a1e2df7ee1afa4db29d818465.jpg temp/1760786478_2998237_1389941599_a732e6f4591a6aed0aff7db86496cfec.jpg temp/1760786478_2998237_1389941545_94681968a424961ac3851224933fc614.jpg temp/1760786478_2998237_1389941529_3817a558bd97213b12ec534dfe3d5ddb.jpg temp/1760786478_2998237_1389941512_bc1a92a8883cae6308491665ddc8a4bf.jpg temp/1760786478_2998237_1389941505_69a522100d1ceefc50a0767eaec341f9.jpg temp/1760786478_2998237_1389941488_6c353cf970f0c79e0088baf1c7e9113e.jpg temp/1760786478_2998237_1389941487_15cde852193fb727d3cc53b1a6317557.jpg temp/1760786478_2998237_1389941482_03271ce88771d672a5bc3d2fd4abcd1a.jpg temp/1760786478_2998237_1389941472_a291277d7d5847943782a734d9379499.jpg temp/1760786478_2998237_1389941469_ca0e392d15cbca69d81d1445f4fe9522.jpg temp/1760786478_2998237_1389941436_349be991bb6c04af556de4a9620d4f30.jpg temp/1760786478_2998237_1389941406_f6e323ea71c6dcd5419af0ef528ea92e.jpg temp/1760786478_2998237_1389941314_905989ca6861b0d9e1b28048c3b1e464.jpg temp/1760786478_2998237_1389941113_f115165d5cfcea11e09ff3800745d7fc.jpg temp/1760786478_2998237_1389941051_7fad67065ad9c6bfc39b13bb21ce865c.jpg temp/1760786478_2998237_1389940984_1d4d80ae57c96e78b2f8e0f6e06e6bfc.jpg temp/1760786478_2998237_1389940025_c5b2d5b688581558401368e9c379daad.jpg temp/1760786478_2998237_1389940004_2962611b28a2a7a9d8b4d61d6cbcf4d2.jpg temp/1760786478_2998237_1389940003_8745f433cf48631317016f4487e6d8e4.jpg temp/1760786478_2998237_1389939960_39a85759fab0441eebc24241024b2332.jpg temp/1760786478_2998237_1389939887_4ffaadb347532af2affd9c1940a2e776.jpg temp/1760786478_2998237_1389939886_13f7d6fea29edb938153062047ac0d67.jpg temp/1760786478_2998237_1389939875_a7420b5b23f429743236a5c1a491c3aa.jpg temp/1760786478_2998237_1389939874_3d24cef04f6015411825b445633b4423.jpg temp/1760786478_2998237_1389939873_e0ca2effa7e8b36a3699e1b5c5164553.jpg temp/1760786478_2998237_1389939811_6741195eddac77cdf2ab18f05b2e130b.jpg temp/1760786478_2998237_1389939764_c5872f2c4a4829256b1399dd75be760e.jpg temp/1760786478_2998237_1389939750_48fd3042dcff5fb8043aa33611805a8d.jpg temp/1760786478_2998237_1389939737_f680bd073785090806f331beed8fc128.jpg temp/1760786478_2998237_1389939724_eb07671bbdcc0f8edef9c9b980875eef.jpg temp/1760786478_2998237_1389939723_ebd739ce655ef088b69fd23cb6228736.jpg temp/1760786478_2998237_1389939694_7fe0877d68799e28b328ee47c3bd9842.jpg temp/1760786478_2998237_1389939630_3e19ddeca578bcceeacffcefd5f65e9d.jpg temp/1760786478_2998237_1389939616_86b51eb796d389ac9f5f9b3df6287660.jpg temp/1760786478_2998237_1389939557_bf36d57e0d88330ce1d264d0da331af5.jpg temp/1760786478_2998237_1389939503_3f7143ddc8d76703b9c2074ce5994af3.jpg temp/1760786478_2998237_1389939458_ee74350c46318368b282ccfa3ab06481.jpg temp/1760786478_2998237_1389939422_fb4b5317018710ebac83371bbe7701d5.jpg temp/1760786478_2998237_1389939338_19bf9266f03ebd28573d8a0cfd6a0efa.jpg temp/1760786478_2998237_1389939311_03bb3e7b1b7e99ceab92e7c81b7e3f9a.jpg temp/1760786478_2998237_1389939285_9b1dd05b96923955af204123cf7822be.jpg temp/1760786478_2998237_1389939215_be82bdae9fdc2e7603ed2c9dd7a29ecf.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2339, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2533, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1389947046, 1389946949, 1389946948, 1389946890, 1389946889, 1389946858, 1389946797, 1389946782, 1389946758, 1389946729, 1389946707, 1389946700, 1389946676, 1389946654, 1389946645, 1389946634, 1389946614, 1389946606, 1389946593, 1389946554, 1389946549, 1389946534, 1389946531, 1389946529, 1389946509, 1389946507, 1389946506, 1389946488, 1389946432, 1389946422, 1389946411, 1389946403, 1389946399, 1389946392, 1389946376, 1389946307, 1389946263, 1389946207, 1389946177, 1389946100, 1389946099, 1389946071, 1389946070, 1389945998, 1389945997, 1389945973, 1389945956, 1389945896, 1389945895, 1389945894, 1389945893, 1389945892, 1389945891, 1389945867, 1389945846, 1389945797, 1389945796, 1389945795, 1389945792, 1389945791, 1389945763, 1389945761, 1389945674, 1389945673, 1389945672, 1389945671, 1389945667, 1389945666, 1389945661, 1389945656, 1389945643, 1389945607, 1389945597, 1389945558, 1389945512, 1389945485, 1389945481, 1389945476, 1389945446, 1389945393, 1389945350, 1389945291, 1389945259, 1389945204, 1389945110, 1389945065, 1389944983, 1389944917, 1389944916, 1389944892, 1389944829, 1389944809, 1389944741, 1389944658, 1389944576, 1389944462, 1389944377, 1389944287, 1389944275, 1389944206, 1389944203, 1389944185, 1389944123, 1389944121, 1389944097, 1389944041, 1389944028, 1389944022, 1389943957, 1389943929, 1389943874, 1389943839, 1389943837, 1389943788, 1389943772, 1389943754, 1389943746, 1389943701, 1389943667, 1389943581, 1389943554, 1389943538, 1389943495, 1389943407, 1389943268, 1389943170, 1389943077, 1389942992, 1389942897, 1389942896, 1389942809, 1389942804, 1389942772, 1389942732, 1389942690, 1389942663, 1389942633, 1389942632, 1389942618, 1389942583, 1389942567, 1389942524, 1389942510, 1389942475, 1389942463, 1389942438, 1389942430, 1389942362, 1389942312, 1389942259, 1389942019, 1389941919, 1389941813, 1389941769, 1389941700, 1389941652, 1389941617, 1389941604, 1389941599, 1389941545, 1389941529, 1389941512, 1389941505, 1389941488, 1389941487, 1389941482, 1389941472, 1389941469, 1389941436, 1389941406, 1389941314, 1389941113, 1389941051, 1389940984, 1389940025, 1389940004, 1389940003, 1389939960, 1389939887, 1389939886, 1389939875, 1389939874, 1389939873, 1389939811, 1389939764, 1389939750, 1389939737, 1389939724, 1389939723, 1389939694, 1389939630, 1389939616, 1389939557, 1389939503, 1389939458, 1389939422, 1389939338, 1389939311, 1389939285, 1389939215] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 22.72210931777954 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 12.820250511169434 time spend to save output : 22.73132038116455 total time spend for step 0 : 35.551570892333984 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.21user 3.52system 1:36.96elapsed 17%CPU (0avgtext+0avgdata 1893660maxresident)k 198560inputs+704032outputs (6major+491428minor)pagefaults 0swaps