python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2963257 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Date 2026-10-10 14:10:30.093850 Current got : datou_id : 533, datou_cur_ids : ['4398658'] with mtr_portfolio_ids : ['31292033'] and first list_photo_ids : [] new path : /proc/2963257/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 149 ; length of list_pids : 149 ; length of list_args : 149 time to download the photos : 50.141899824142456 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat Oct 10 14:11:20 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2026-10-10 14:11:23.344513: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2026-10-10 14:11:23.373220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-10-10 14:11:23.373486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-10-10 14:11:23.375607: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-10-10 14:11:23.377622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-10-10 14:11:23.377938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-10-10 14:11:23.379770: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-10-10 14:11:23.380766: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-10-10 14:11:23.394721: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-10-10 14:11:23.396300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-10-10 14:11:23.396750: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2026-10-10 14:11:23.425373: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492960000 Hz 2026-10-10 14:11:23.426805: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fdf54000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2026-10-10 14:11:23.426824: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2026-10-10 14:11:23.636349: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x15efe230 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2026-10-10 14:11:23.636393: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2026-10-10 14:11:23.637445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-10-10 14:11:23.637548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-10-10 14:11:23.637577: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-10-10 14:11:23.637603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-10-10 14:11:23.637629: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-10-10 14:11:23.637655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-10-10 14:11:23.637681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-10-10 14:11:23.637707: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-10-10 14:11:23.639285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-10-10 14:11:23.639808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-10-10 14:11:23.640942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-10-10 14:11:23.640967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-10-10 14:11:23.640981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-10-10 14:11:23.642494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3691 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:14 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:27 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:28 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2026-03-04 13:32:04 create time in s3 : 2026-03-04 12:24:36 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.4434754848480225 time used to load_weights : 0.20679926872253418 0it [00:00, ?it/s] 13it [00:00, 122.08it/s] 26it [00:00, 121.30it/s] 39it [00:00, 121.76it/s] 52it [00:00, 122.01it/s] 65it [00:00, 121.24it/s] 78it [00:00, 121.51it/s] 91it [00:00, 121.84it/s] 104it [00:00, 121.83it/s]temp/1791634230_2963257_1420246939_030c79d028e1f32cdca8edc58e1f0977.jpg temp/1791634230_2963257_1420246937_a5b3e408ff5a4e4fc79bef0fb119f177.jpg temp/1791634230_2963257_1420246935_903c50bb4f71ac2760f3d906ce99e49e.jpg temp/1791634230_2963257_1420246934_f85e7479a63ce9a225f730c4ced2772c.jpg temp/1791634230_2963257_1420246933_59f36d1cf205cc0e3f6a53200e8be0ad.jpg temp/1791634230_2963257_1420246932_8dec5b0c1bf8b59e49d79835ef410018.jpg temp/1791634230_2963257_1420246923_f162fff255a67a3af29e1b2c195521df.jpg temp/1791634230_2963257_1420246919_4f79c578348efd08e9d0fa7db12345ed.jpg temp/1791634230_2963257_1420246918_4b1991d09a899a1418fc250326a4ca87.jpg temp/1791634230_2963257_1420246917_7f4863d0df73fdb93e4aafcc7e2e5fd5.jpg temp/1791634230_2963257_1420246916_30a1c69324e063aa85d4dcbc7f632b6a.jpg temp/1791634230_2963257_1420246915_bd5f09d12072bede8dfca302eb08c11c.jpg temp/1791634230_2963257_1420244969_c10700c5f8c40ec10c9e7de4d2db077b.jpg temp/1791634230_2963257_1420244910_6de4374d03bb75af7ff783f60df91a1b.jpg temp/1791634230_2963257_1420244880_5ab09fc276524c94a81d20e7d0b978ee.jpg temp/1791634230_2963257_1420244879_68f242174c57fb3641f9a6d412ebe00d.jpg temp/1791634230_2963257_1420244878_68e26b98fc20b1e9ee9f9511c6993f68.jpg temp/1791634230_2963257_1420244877_39961489e285335358f23071acbd564e.jpg temp/1791634230_2963257_1420244476_eaa0d1cf9d8536f97d6707a709bf79fe.jpg temp/1791634230_2963257_1420244475_0c4a625bcb49adba3adc35d92bd09675.jpg temp/1791634230_2963257_1420244474_a9cb2807dfcf259b4d1c0b854c028b1c.jpg temp/1791634230_2963257_1420244473_894bbd4982eb95c8c50840f77f1b8063.jpg temp/1791634230_2963257_1420244472_c1312a8a745367e69728a69051f71bff.jpg temp/1791634230_2963257_1420244471_42819fdcba88e8cc2e9c7441254fa420.jpg temp/1791634230_2963257_1420241844_6f4df4c7553223ce8149f69f2ad67740.jpg temp/1791634230_2963257_1420241842_108600579c9009886f69bdc2d0e29055.jpg temp/1791634230_2963257_1420241841_f9b9ddfc2046b061d2feb1074c552f03.jpg temp/1791634230_2963257_1420241840_d27281fbdf0dbc9d663289a85d617c68.jpg temp/1791634230_2963257_1420241839_936fdba27fd6d74383f63d0a54a06cad.jpg temp/1791634230_2963257_1420241838_aa87595ac2b8fec1585d70a5010f7d1a.jpg temp/1791634230_2963257_1420241828_0d60607afd3dc23e57ed4dae3b98b411.jpg temp/1791634230_2963257_1420241826_fdfaec6c2e7846145647b8ef359696fe.jpg temp/1791634230_2963257_1420241825_2a7f83703adbbf118552fdd3b749f4f8.jpg temp/1791634230_2963257_1420241822_198cfccc540fe42a8b293753af94b1bd.jpg temp/1791634230_2963257_1420241797_a049db1655be424e94e145172d6b33a6.jpg temp/1791634230_2963257_1420241778_644f0f4a7b66819d3abd8aa0610159a5.jpg temp/1791634230_2963257_1420241488_5277e06c8a5c65b6b76fa4d3ffbff83d.jpg temp/1791634230_2963257_1420241487_86d259766d47374321997166b1effccf.jpg temp/1791634230_2963257_1420241486_66d342eab68320d8abfeb731e0bec0c1.jpg temp/1791634230_2963257_1420241485_ea816b042fc21a59db93cb227e4c7d89.jpg temp/1791634230_2963257_1420240599_ce3b6bd0cda7c483f125b95878484855.jpg temp/1791634230_2963257_1420240544_182166197ef60c103f66e4d15156b415.jpg temp/1791634230_2963257_1420240492_17f469a4ece48dc7cb6c030227d9aebc.jpg temp/1791634230_2963257_1420240438_fd15bb21a112280c81a1f52313623925.jpg temp/1791634230_2963257_1420240385_8386536f09de17cb4281055557281a41.jpg temp/1791634230_2963257_1420240329_49f20d5bcafc84b3f4ffbb7a5ff266f0.jpg temp/1791634230_2963257_1420237965_863bef3d461ef48bfa48b0f96c346319.jpg temp/1791634230_2963257_1420237933_a3b759e9fcd9b8339909e71ea3ac0428.jpg temp/1791634230_2963257_1420237861_ecdfdc9ac921502e95cd7f44820db3b4.jpg temp/1791634230_2963257_1420237776_54f620d4b8c254733b29b0fba34a821a.jpg temp/1791634230_2963257_1420237671_1c906abd1beab83ea56352a1925079df.jpg temp/1791634230_2963257_1420237561_3a777cb41bdc5f0f3cc0166139fe4e2d.jpg temp/1791634230_2963257_1420236105_d5404b1927431979545d36e5062e80a7.jpg temp/1791634230_2963257_1420236055_b378fd114d40581f5259a38aff4f06f9.jpg temp/1791634230_2963257_1420236004_975367641a167be5b856b7b05323a0c9.jpg temp/1791634230_2963257_1420235948_7a53dfbb0e1c0346b22538fde3a13159.jpg temp/1791634230_2963257_1420235894_bce446a59f9e6a10330b88494d7aac37.jpg temp/1791634230_2963257_1420235843_a27f3e56dc8b1e70ec3ddea606ebce10.jpg temp/1791634230_2963257_1420234347_755d67702a6945570e750397f13771ac.jpg temp/1791634230_2963257_1420234346_974bc8bab57bf25a054b59c7a3c3a409.jpg temp/1791634230_2963257_1420234344_484590f7b4e1ff7172130ee47a0f8c8e.jpg temp/1791634230_2963257_1420234342_44c26de202e1c14e406dc1d2f28c4049.jpg temp/1791634230_2963257_1420234341_4a139e6b3d39cd9beb4c70fb13336c24.jpg temp/1791634230_2963257_1420234340_ead90cd431c4a0f85ccec468e3c33e17.jpg temp/1791634230_2963257_1420234329_b683529d15e6d1134739cda19bba3739.jpg temp/1791634230_2963257_1420234327_459500513460cd6566550b1c17ca3e76.jpg temp/1791634230_2963257_1420234326_e54c0dbcc10ae243d5d7cae8cdb077c7.jpg temp/1791634230_2963257_1420234325_e3c8c89867fb670312de792c0bea1a64.jpg temp/1791634230_2963257_1420234324_d2dff8233d0d6c0476c40d1d33daa1d5.jpg temp/1791634230_2963257_1420234323_0fd11be7c8cc19892448cdce748a0c69.jpg temp/1791634230_2963257_1420234313_aaa8fcfcdec4235d303885789b30e14c.jpg temp/1791634230_2963257_1420234310_1f158ce3497886e2c2d29fa019b8cd1a.jpg temp/1791634230_2963257_1420234309_5cc86d851819849974ffb75999a33d4b.jpg temp/1791634230_2963257_1420234308_4950ac57a4f566a1020c26a963dc3ece.jpg temp/1791634230_2963257_1420234307_eefbffd8b0b28994da974e0e83b54b03.jpg temp/1791634230_2963257_1420234306_2d119bb83e1d165be8c442fe91c2bb4e.jpg temp/1791634230_2963257_1420234304_dc15d850b620bde9d3576d527ad57859.jpg temp/1791634230_2963257_1420234303_5869357c5049db4ca039bd9322864705.jpg temp/1791634230_2963257_1420234302_4c19d28a58f0e87acd7cb589e4ef2c09.jpg temp/1791634230_2963257_1420234301_ea7b84d02c726ddf7577e015be331d89.jpg temp/1791634230_2963257_1420234284_0f0ff2240ea47ff06496f409ace2ffdb.jpg temp/1791634230_2963257_1420234283_fd29a3f4e344253d2624d03756696515.jpg temp/1791634230_2963257_1420234282_ee4805c2851295a9f6ad49d1dfa15cf2.jpg temp/1791634230_2963257_1420234281_0c32a591bfac3bd57adc29a6d1964891.jpg temp/1791634230_2963257_1420234280_357664e45a5932081e842eeea1d88a31.jpg temp/1791634230_2963257_1420234279_e27af5191037ee60fe6f1f54f44dad8c.jpg temp/1791634230_2963257_1420233895_73ccd2c4285ed0bc158b28e178996366.jpg temp/1791634230_2963257_1420233894_95df2b6c8e4bc4f10fdd2a7190e8ff12.jpg temp/1791634230_2963257_1420233893_7c8ec58b6005c6e6eb4487f1953207dd.jpg temp/1791634230_2963257_1420233892_baf25bc0c2a2091183720670697dda4d.jpg temp/1791634230_2963257_1420233891_9cd86028afdc478c2d9799dc2f96f391.jpg temp/1791634230_2963257_1420233890_ce3606042cca862234224f659cd7b6dc.jpg temp/1791634230_2963257_1420232473_d640966e74862392c728d80d6135df0f.jpg temp/1791634230_2963257_1420232416_6fffa30e252088e543bbb75248ca088e.jpg temp/1791634230_2963257_1420232339_c18d6b3ea1a57c82d49281b35871ca91.jpg temp/1791634230_2963257_1420232324_69bda63ccebefbbfea5ed3268854b630.jpg temp/1791634230_2963257_1420232322_e2ae709cef2e5bb07a77881b4ad21637.jpg temp/1791634230_2963257_1420232304_d31aebcb3bb4064ca4ca69a62e5062c0.jpg temp/1791634230_2963257_1420231428_03686f5867adb205d54140577efb10e8.jpg temp/1791634230_2963257_1420231427_6e2203d6d78ef1d3e4cc8c639ef0281d.jpg temp/1791634230_2963257_1420231426_dd5bff844f077ef565d2673ca4db0558.jpg temp/1791634230_2963257_1420231413_f2bd3aa1f173d162f75411e6137191a4.jpg temp/1791634230_2963257_1420231412_aee71eb01f564f8e77b2cc9ad1bcb1a7.jpg temp/1791634230_2963257_1420231376_994a0ceacfb21361fc666fb0ce0749d2.jpg temp/1791634230_2963257_1420230115_ac426186d40d1cde92ea5fbffd6f7d5e.jpg temp/1791634230_2963257_1420230113_75b2d602ac459faf676cb137904fa881.jpg temp/1791634230_2963257_1420230112_062012c3bcd6a8d0494e429250d11a1d.jpg temp/1791634230_2963257_1420230111_8b31b14a3550ffdb3d0953fd5ffd4b95.jpg temp/1791634230_2963257_1420230110_4f5cacc52c57058cbab3a09fd90b7727.jpg temp/1791634230_2963257_1420230109_33bf36a877fba39f508e11fb56d2a7cc.jpg temp/1791634230_2963257_1420229751_16eefcfe04e69d8309479c816c16789a.jpg temp/1791634230_2963257_1420229750_cb8f7107ba944a0da8ed31eb3f6bc25d.jpg temp/1791634230_2963257_1420229749_879f732478eefda376c0fa680fd03220.jpg 117it [00:00, 121.32it/s] 130it [00:01, 121.61it/s] 143it [00:01, 120.88it/s] 149it [00:01, 120.70it/s]2026-10-10 14:11:35.229162: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-10-10 14:11:35.690433: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-10-10 14:11:36.545651: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2026-10-10 14:11:36.551330: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1791634230_2963257_1420229748_5e601a169ed11a5c010a4ec665a06255.jpg temp/1791634230_2963257_1420229747_7c6583c11ebfb4912aee8ad6e110763b.jpg temp/1791634230_2963257_1420229746_383558fd3a98ecf62154e371cc2c423c.jpg temp/1791634230_2963257_1420229743_0349f83ed357a4267a7270f13ba8d527.jpg temp/1791634230_2963257_1420229742_da9e673e7da37c59bb0c2bc3c72baf10.jpg temp/1791634230_2963257_1420229741_7ab97cf1d92923c9cc82eceed17b79a0.jpg temp/1791634230_2963257_1420227404_63d60980326791ea57a2c98863a3dada.jpg temp/1791634230_2963257_1420227402_9f11246b01b31c8315e27cad37c92f84.jpg temp/1791634230_2963257_1420227401_49af9e9dc13d8a78207431d582a3f944.jpg temp/1791634230_2963257_1420227400_f3a9462ba4222d8a87d878c3340fdcef.jpg temp/1791634230_2963257_1420227399_bef6840bf9654339e919fe5f7ba3373d.jpg temp/1791634230_2963257_1420227398_0ebd73cd257c30dbf78b4fa07e7bcb4a.jpg temp/1791634230_2963257_1420226902_abe83e452cf815c5365ab8de28c1bd77.jpg temp/1791634230_2963257_1420226901_26bb8d117f3e0178e6c529f57445f9b5.jpg temp/1791634230_2963257_1420226899_9524e0bd34bada8a25a7e1f02adf39d4.jpg temp/1791634230_2963257_1420226898_ab1fd50257fadc2daba8143d1bbc33bf.jpg temp/1791634230_2963257_1420226897_c338ba50b2f4ff917511ecb1bbe1bcc3.jpg temp/1791634230_2963257_1420226896_3f4eac9b351570bbc538cf147fd2dbbf.jpg temp/1791634230_2963257_1420226879_e264dd940f641bb3e0823c38330dfe03.jpg temp/1791634230_2963257_1420226877_b28e1a7f5c3e978556429c9fd611f471.jpg temp/1791634230_2963257_1420226876_67d2126c2a527518406bb79a1a57ed07.jpg temp/1791634230_2963257_1420226875_63e82758de17736b73fc6f182d114733.jpg temp/1791634230_2963257_1420226874_3d9859e3db0734d5df02ee0343225b21.jpg temp/1791634230_2963257_1420226873_75fc250fdf40eb7eb1c3b84b40566778.jpg temp/1791634230_2963257_1420225966_d2bd4f764c128386adc83e4d3f72ea21.jpg temp/1791634230_2963257_1420225921_8e070448646d878c3fde30b09bcdd1c5.jpg temp/1791634230_2963257_1420225879_c88bdd9c0f3ef4cdbe133707b492687c.jpg temp/1791634230_2963257_1420225824_a2bb60501291c40b7f9a9f3326461233.jpg temp/1791634230_2963257_1420225747_1f9e4e13a5ef86a02b7af175654abe16.jpg temp/1791634230_2963257_1420225695_75d7c68bcb62d43dac715ce019b4413c.jpg temp/1791634230_2963257_1420221488_44f07500fd91907160cf0d44d7c4b44b.jpg temp/1791634230_2963257_1420221379_914497cac19d02ee923f44d6c3f2e631.jpg temp/1791634230_2963257_1420221221_6fdc11c4eef8e2b0c9f9e24af5d79706.jpg temp/1791634230_2963257_1420221177_2e7695f3e5ff6ff9a25861b7f6b67ecd.jpg temp/1791634230_2963257_1420221122_d4f730dd3103031a8fb275423efb4e70.jpg temp/1791634230_2963257_1420221072_22dc1948fc755e8c3f691658831f5770.jpg Found 149 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2340, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2534, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3171, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1420246939, 1420246937, 1420246935, 1420246934, 1420246933, 1420246932, 1420246923, 1420246919, 1420246918, 1420246917, 1420246916, 1420246915, 1420244969, 1420244910, 1420244880, 1420244879, 1420244878, 1420244877, 1420244476, 1420244475, 1420244474, 1420244473, 1420244472, 1420244471, 1420241844, 1420241842, 1420241841, 1420241840, 1420241839, 1420241838, 1420241828, 1420241826, 1420241825, 1420241822, 1420241797, 1420241778, 1420241488, 1420241487, 1420241486, 1420241485, 1420240599, 1420240544, 1420240492, 1420240438, 1420240385, 1420240329, 1420237965, 1420237933, 1420237861, 1420237776, 1420237671, 1420237561, 1420236105, 1420236055, 1420236004, 1420235948, 1420235894, 1420235843, 1420234347, 1420234346, 1420234344, 1420234342, 1420234341, 1420234340, 1420234329, 1420234327, 1420234326, 1420234325, 1420234324, 1420234323, 1420234313, 1420234310, 1420234309, 1420234308, 1420234307, 1420234306, 1420234304, 1420234303, 1420234302, 1420234301, 1420234284, 1420234283, 1420234282, 1420234281, 1420234280, 1420234279, 1420233895, 1420233894, 1420233893, 1420233892, 1420233891, 1420233890, 1420232473, 1420232416, 1420232339, 1420232324, 1420232322, 1420232304, 1420231428, 1420231427, 1420231426, 1420231413, 1420231412, 1420231376, 1420230115, 1420230113, 1420230112, 1420230111, 1420230110, 1420230109, 1420229751, 1420229750, 1420229749, 1420229748, 1420229747, 1420229746, 1420229743, 1420229742, 1420229741, 1420227404, 1420227402, 1420227401, 1420227400, 1420227399, 1420227398, 1420226902, 1420226901, 1420226899, 1420226898, 1420226897, 1420226896, 1420226879, 1420226877, 1420226876, 1420226875, 1420226874, 1420226873, 1420225966, 1420225921, 1420225879, 1420225824, 1420225747, 1420225695, 1420221488, 1420221379, 1420221221, 1420221177, 1420221122, 1420221072] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 149 time used for this insertion : 0.1143953800201416 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 16.140981435775757 time spend to save output : 0.12310624122619629 total time spend for step 0 : 16.264087677001953 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.72user 8.07system 1:12.02elapsed 31%CPU (0avgtext+0avgdata 2598812maxresident)k 512808inputs+4479248outputs (932major+793815minor)pagefaults 0swaps