python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2781648 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 1421>200 please execute split_portfolio.py -i 27896762 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3957315'] with mtr_portfolio_ids : ['27896762'] and first list_photo_ids : [] new path : /proc/2781648/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 39.201664209365845 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat Oct 18 10:21:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-10-18 10:21:34.687334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-10-18 10:21:34.718725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-18 10:21:34.719245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 10:21:34.722313: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 10:21:34.724445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-18 10:21:34.725266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-18 10:21:34.728834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-18 10:21:34.730161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-18 10:21:34.738232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 10:21:34.740064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-18 10:21:34.740564: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-10-18 10:21:34.766486: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-10-18 10:21:34.768309: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fac84000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-10-18 10:21:34.768339: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-10-18 10:21:34.969034: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1fb314e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-10-18 10:21:34.969085: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-10-18 10:21:34.970012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-18 10:21:34.970090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 10:21:34.970119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 10:21:34.970145: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-18 10:21:34.970171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-18 10:21:34.970205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-18 10:21:34.970231: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-18 10:21:34.970258: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 10:21:34.971722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-18 10:21:34.971790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 10:21:34.972650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-10-18 10:21:34.972675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-10-18 10:21:34.972689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-10-18 10:21:34.974193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3441 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.807405233383179 time used to load_weights : 0.21117401123046875 0it [00:00, ?it/s] 92it [00:00, 911.56it/s]temp/1760775652_2781648_1389931725_e06acfa37773dba044a3cae632602f8e.jpg temp/1760775652_2781648_1389931705_3a30b0342066c0e5d5895556299c6204.jpg temp/1760775652_2781648_1389931662_552ed04421fab49d61aed62cfc18a983.jpg temp/1760775652_2781648_1389931636_d8e1c2f719bd1aa6e791b3ce818846ed.jpg temp/1760775652_2781648_1389931632_ff88f9d1b0acad5b675adc4cd5308f52.jpg temp/1760775652_2781648_1389931570_89819af52dcaf230249e0d3039f4adfe.jpg temp/1760775652_2781648_1389931562_ec98dd6de3dfeabb2101f32a5c7f2352.jpg temp/1760775652_2781648_1389931557_b1c5d8e4c3a568e59fb24767810c0538.jpg temp/1760775652_2781648_1389931498_419f8060963b62cae1652b5f15283876.jpg temp/1760775652_2781648_1389931495_b38f32bfa75a1c2eeb6bf3ef49c1902f.jpg temp/1760775652_2781648_1389931492_174b71402fed0934de2c0e019b965c07.jpg temp/1760775652_2781648_1389931490_4158bfd97ac7b7531ad776e041895d9f.jpg temp/1760775652_2781648_1389931432_d8a85b1299b909cf040c4c1502e5ef64.jpg temp/1760775652_2781648_1389931431_3c48f2ed5dedb536107b0446d8dff4ca.jpg temp/1760775652_2781648_1389931426_ed2f53c6c0b759428b081bcd0aca6a8b.jpg temp/1760775652_2781648_1389931420_5603b8084830a353a3dbcc6509047d5c.jpg temp/1760775652_2781648_1389931390_8acdf4b8eb23aac5677c84bd7a9dbe4b.jpg temp/1760775652_2781648_1389931377_5eeb13c9d8c0c40582530df443ede19b.jpg temp/1760775652_2781648_1389931372_e4f1b30a35643e1eb142538ac6f645c4.jpg temp/1760775652_2781648_1389931350_0880bd94e3b707a4deb3ba2ef2559317.jpg temp/1760775652_2781648_1389931343_528e9c102b1bf88da7d85ba3b2f96441.jpg temp/1760775652_2781648_1389931342_19bedda54bbd380582154cb2762225ca.jpg temp/1760775652_2781648_1389931340_1de86f0b75ac63bd1651dd46509d33c9.jpg temp/1760775652_2781648_1389931311_29b137b0c869bc9cbfbea03d0e8a527c.jpg temp/1760775652_2781648_1389931302_c44b4f22c6ed85885d994cd07c869810.jpg temp/1760775652_2781648_1389931273_c2d9a4259e1ae8db77895d4fbb4886b3.jpg temp/1760775652_2781648_1389931262_491333178295ffd999fefcbd5cdfb044.jpg temp/1760775652_2781648_1389931259_a7d279a0eebfa377a7ffadf18db5ca9a.jpg temp/1760775652_2781648_1389931255_dc2747a657ea4345ddcade19a2ee21af.jpg temp/1760775652_2781648_1389931239_d5e7cc0d27ccfb2f24168ebd3e88a2ed.jpg temp/1760775652_2781648_1389931212_7005c6a5de15ab991db2a4111c76734b.jpg temp/1760775652_2781648_1389931166_16fd03bdc47ba56da85d55dab58d02cc.jpg temp/1760775652_2781648_1389931149_54f14c63222229efa1c34532b52ee654.jpg temp/1760775652_2781648_1389931137_fdd5af5f58a7606efabc2f8be3cf6e3d.jpg temp/1760775652_2781648_1389931104_d6636afab7c51b7163da0185739f7c73.jpg temp/1760775652_2781648_1389931082_a32f6241b330cb5cf46297d73e04c85f.jpg temp/1760775652_2781648_1389931074_d0e9e8fdc2b9f35bc0583085abefebfd.jpg temp/1760775652_2781648_1389931056_d27afbebd82f008bcb7e32eaadc3f67a.jpg temp/1760775652_2781648_1389931049_fed3a2c0709331d9dffc68e353ddfdbb.jpg temp/1760775652_2781648_1389931033_9a45031627adee20076dc41c9103d9a9.jpg temp/1760775652_2781648_1389931019_89b2cbac943ebf296bf1bc16350b3af8.jpg temp/1760775652_2781648_1389930996_a6273a9f43145eed876ca69a80a4df97.jpg temp/1760775652_2781648_1389930986_7a623ee9c3df4e0f676651b9c8ae9695.jpg temp/1760775652_2781648_1389930982_ee96fb7180bf4ee48c6a21b8ab7d1c9d.jpg temp/1760775652_2781648_1389930981_3aa9baf96f1d8034f036f8fc8aa49609.jpg temp/1760775652_2781648_1389930956_bfd509f9ed4d88bbcfaa110a25f5e69f.jpg temp/1760775652_2781648_1389930927_59f77b38a1fdbfcc7eff7234a88e68c5.jpg temp/1760775652_2781648_1389930925_db84f39e40a29943730956053a4d706e.jpg temp/1760775652_2781648_1389930920_66ef7ed17b3ca07154cd8441e91ce7dc.jpg temp/1760775652_2781648_1389930897_0cfcc500c0d7a7d9cbf3ad9369e77be8.jpg temp/1760775652_2781648_1389930862_b2bec61e0c2d4fb3c64cc77fbe3ae7cc.jpg temp/1760775652_2781648_1389930861_fd9a215235f8eb2736c54e6b30378636.jpg temp/1760775652_2781648_1389930860_88a1ac6f940948367757122321bd5556.jpg temp/1760775652_2781648_1389930817_6a7cb832120df17547637f6a80be4659.jpg temp/1760775652_2781648_1389930806_31433be92e7f6ad917e3d702c4f2a497.jpg temp/1760775652_2781648_1389930796_ecc8000e79a812f5afff01374523d36f.jpg temp/1760775652_2781648_1389930795_e2ceea2c67ba5513ba000d283b093294.jpg temp/1760775652_2781648_1389930788_39bb032d0ee2f9a21eaab887f6e2109c.jpg temp/1760775652_2781648_1389930747_50c686144db372d625463ee6f59daa61.jpg temp/1760775652_2781648_1389930745_691315ad37c83442b9355bb8092a22a1.jpg temp/1760775652_2781648_1389930736_52b2063600688c3af909390ec342686f.jpg temp/1760775652_2781648_1389930732_9622c5f7cd3e43dad6d48a293b20f361.jpg temp/1760775652_2781648_1389930721_2ccf6264acadd98e25ff360b342be9ae.jpg temp/1760775652_2781648_1389930715_03461a9268adfb1c8fc5212e4dd72f97.jpg temp/1760775652_2781648_1389930683_29c66f6f1f1adfc07fb381ed726dcbe8.jpg temp/1760775652_2781648_1389930674_cf8a24f71b2914e0fd759a89377026b7.jpg temp/1760775652_2781648_1389930658_8e61d03a88868077fd6e45c8b0eeed5c.jpg temp/1760775652_2781648_1389930657_26823bf38aae74ca528392adda41fc8e.jpg temp/1760775652_2781648_1389930644_3dbb4dde5b6934a0cb384011cb8a8c4d.jpg temp/1760775652_2781648_1389930620_91c0a3380e1581993a7d267c9754c4db.jpg temp/1760775652_2781648_1389930596_7eb9801ff91d22b775b7727136b5d8d0.jpg temp/1760775652_2781648_1389930590_4a43f6cb35f1515122432135b745eaf9.jpg temp/1760775652_2781648_1389930589_392b396b9d0dae1b964a598b4edd7eb4.jpg temp/1760775652_2781648_1389930579_08426897658fa1d83f99bd53a3609072.jpg temp/1760775652_2781648_1389930555_3321143ff57c9d50f7c4d688629b4f5e.jpg temp/1760775652_2781648_1389930540_8abaa69f56290808f4397843eff69ee5.jpg temp/1760775652_2781648_1389930534_183a6be42f0f4ee0094d497f2909c38a.jpg temp/1760775652_2781648_1389930528_421ce9ae45cae8646a368b1da1c1c9d2.jpg temp/1760775652_2781648_1389930519_74fa1d473f2001aae32da9d228d18df8.jpg temp/1760775652_2781648_1389930514_1a0e50b03f320aa649b563873d76e0eb.jpg temp/1760775652_2781648_1389930493_a5e8a94c535be5c25cd5b4e37efff8c9.jpg temp/1760775652_2781648_1389930487_8efb2bdc459b25ebdc5e9037d5a2d3eb.jpg temp/1760775652_2781648_1389930472_b4611c9c59988a25ce7f4a43e723ba5f.jpg temp/1760775652_2781648_1389930463_3dcf6c9cd3f9a745db1a8333bd4290f8.jpg temp/1760775652_2781648_1389930443_9c9501a8fc2891e07931ae8538512c7a.jpg temp/1760775652_2781648_1389930434_37f9b00d8126068148713a5c5b449fd7.jpg temp/1760775652_2781648_1389930419_959611ec288746637679389a30ceb73a.jpg temp/1760775652_2781648_1389930403_a6ddd85d56b460324a8627b8511d78ae.jpg temp/1760775652_2781648_1389930398_e884f043994aa2dd2ecb240163ae80ba.jpg temp/1760775652_2781648_1389930390_9bf25d2ff4549728cfe8d0ece41dde5f.jpg temp/1760775652_2781648_1389930381_0aedc6a19b5745304f98095fa0450cc2.jpg temp/1760775652_2781648_1389930363_7e367eb5faf78932608cbd34c23043df.jpg temp/1760775652_2781648_1389930349_7a1ddd2fc0661c990d516bd07664ed4d.jpg temp/1760775652_2781648_1389930336_b0b829165a90031c706e752c640196e8.jpg temp/1760775652_2781648_1389930331_99dc01d5b18de85bc80b2791db6a8cdc.jpg temp/1760775652_2781648_1389930284_de9b28d0c76480b776ff17fa7648914f.jpg temp/1760775652_2781648_1389930273_9409c25d27e5755f4cb823b2f614306c.jpg temp/1760775652_2781648_1389930225_3a423c79995996dbdde9a8c031c86f2a.jpg temp/1760775652_2781648_1389930215_09158ffdde059983f104e727aa332e13.jpg temp/1760775652_2781648_1389930213_c9f35841f69ceb026752031c85149f74.jpg temp/1760775652_2781648_1389930182_a9446e70881c62976bebe3fd387786bc.jpg temp/1760775652_2781648_1389930175_7e8a5b59ee15844f74a12bc9c7606251.jpg temp/1760775652_2781648_1389930169_193d84f760a35605d7bd68036491bac2.jpg temp/1760775652_2781648_1389930161_7698e7cec1c07f3c22c248726e05380e.jpg temp/1760775652_2781648_1389930137_b32eb29074c6e5326921101082cdddc5.jpg temp/1760775652_2781648_1389930130_e6a2b0ab7fc79486edad5fbba34635e0.jpg temp/1760775652_2781648_1389930129_1543c48e941a12b5b92524d654b6686b.jpg temp/1760775652_2781648_1389930126_7287ac8666e877b03878f621233702b6.jpg temp/1760775652_2781648_1389930117_37266ea084c23c308dc5b5f429fe5c0a.jpg temp/1760775652_2781648_1389930114_f7c442950fdbcf38a402c72730637b84.jpg temp/1760775652_2781648_1389930102_d53bdba96c51045c1f07f52b88606910.jpg temp/1760775652_2781648_1389930101_12ef4e47019e6cda8508a771dd9f5a29.jpg temp/1760775652_2781648_1389930095_922e10282908480bb03c5eb1a04e4057.jpg 189it [00:00, 942.89it/s] 200it [00:00, 943.53it/s]2025-10-18 10:21:45.569605: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 10:21:45.732345: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-10-18 10:21:45.814665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 10:21:45.820156: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-10-18 10:21:45.822413: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1760775652_2781648_1389930094_f31f4ef9c70d2d10680ac4c84307b3bc.jpg temp/1760775652_2781648_1389930078_f23471b78eb53cdafd03074a3da16161.jpg temp/1760775652_2781648_1389930069_893d6e2f4fcc670c0b6964675fffc279.jpg temp/1760775652_2781648_1389930068_4fbf8f7f7590050b38233c73fba2182e.jpg temp/1760775652_2781648_1389930062_b61e9d4c04de1742569cd8f3ea528bd3.jpg temp/1760775652_2781648_1389930061_a44a2cd585ab85923bb8acb40d98c586.jpg temp/1760775652_2781648_1389930051_5892e820774313e65542b09220d44088.jpg temp/1760775652_2781648_1389930050_06dd5b989289271aca4f7cd2af5f8632.jpg temp/1760775652_2781648_1389930046_713c97265f595e7a29bcfb336e995239.jpg temp/1760775652_2781648_1389930034_630367cacdea0339b9ebe26853eb6f16.jpg temp/1760775652_2781648_1389930028_6d4a1247fdbf472fefc8330d106b9a8f.jpg temp/1760775652_2781648_1389930019_081467b609c2798b7b9212d0c201c6d6.jpg temp/1760775652_2781648_1389930017_223232b8f597794e52c060e4ac774daa.jpg temp/1760775652_2781648_1389930014_ffd543527fc59e50c79688090c2f85c6.jpg temp/1760775652_2781648_1389930013_7d53e242ec9b69c232a8720091112319.jpg temp/1760775652_2781648_1389929995_e03a144f7b83fd877e1c548eac45ee43.jpg temp/1760775652_2781648_1389929986_9606751eb6d2bffc5e0babf984614e35.jpg temp/1760775652_2781648_1389929979_5da27f8d102d6ea1118ae10d092ee25b.jpg temp/1760775652_2781648_1389929975_c6e0120a35aeefe95888a5d2c2c5332b.jpg temp/1760775652_2781648_1389929969_d1b12a6aa5ece3a9725ff81047abe3e9.jpg temp/1760775652_2781648_1389929950_cfcbea0132e19a1a65a2ae3a6ca38374.jpg temp/1760775652_2781648_1389929935_9ceff2c2548fcc6d1363a4968b214f7b.jpg temp/1760775652_2781648_1389929910_34380cbe9500a16f485bdb418c1b15f3.jpg temp/1760775652_2781648_1389929899_ed540cf3bdddb966f4d6bc16ee364a1e.jpg temp/1760775652_2781648_1389929879_7c14aa9d98e8dfbcbf2a05d7677636dd.jpg temp/1760775652_2781648_1389929831_85cbe9e5aa95b98dffeb2ae8b60d7cf2.jpg temp/1760775652_2781648_1389929820_1be6b791779020ddaef448c201c03dca.jpg temp/1760775652_2781648_1389929808_94819f1ed95accd464761231d6b8160a.jpg temp/1760775652_2781648_1389929787_e607066a1aa560230d49ad7de44346b7.jpg temp/1760775652_2781648_1389929786_5ad331436c574a354593da85b2d3b1b1.jpg temp/1760775652_2781648_1389929737_c076aae70239e89f1204faceb4ef69f3.jpg temp/1760775652_2781648_1389929714_95fb600e86c0569f77fdbb19fc83ad8e.jpg temp/1760775652_2781648_1389929693_5b952c81f75b415df834bedaaed56f78.jpg temp/1760775652_2781648_1389929683_43fe4d75ebcc5af1f10f688cd6efa980.jpg temp/1760775652_2781648_1389929663_3b8e515df3e2b69c38dbc76d4dff5abe.jpg temp/1760775652_2781648_1389929660_ed586f807a1e0c44d60b8c4e1b6dce1e.jpg temp/1760775652_2781648_1389929654_3fe33d129fd3b28f257d3ae3b6b09a1e.jpg temp/1760775652_2781648_1389929630_560e603beffd2b257e05c29fead6b946.jpg temp/1760775652_2781648_1389929619_851db5888d39e0767921d2c3596c7447.jpg temp/1760775652_2781648_1389929618_c6dd137df1c2105de7acf5deafca072f.jpg temp/1760775652_2781648_1389929601_eb388b8250ff39128303098e00c61177.jpg temp/1760775652_2781648_1389929584_35e91a0bbab9bfe3f5fa446e3c83f46d.jpg temp/1760775652_2781648_1389929583_ce3c1ea6a531caade728f785323f4fe0.jpg temp/1760775652_2781648_1389929568_f7f9307fd5be593ab5a86ac531187ad8.jpg temp/1760775652_2781648_1389929567_5f5bd03cd95d9b80b5b931345902faa7.jpg temp/1760775652_2781648_1389929554_60923ea196d4d17a0d474b00b8161e6a.jpg temp/1760775652_2781648_1389929546_3cca632b48712c8c6fa62abadf6b3c82.jpg temp/1760775652_2781648_1389929540_d1266a1803f9f9ea929bb266163c5091.jpg temp/1760775652_2781648_1389929538_357775b6c47d325f3745b707025b4ba7.jpg temp/1760775652_2781648_1389929534_06d15dca4801f2220a06c606103f5a42.jpg temp/1760775652_2781648_1389929518_3cb838e72a30efad739169e7a6757225.jpg temp/1760775652_2781648_1389929494_bc0dbc258dfca8ea419f67cc7d428ed5.jpg temp/1760775652_2781648_1389929466_5f525a9fbb309d0c7b3d85dbefa62cd2.jpg temp/1760775652_2781648_1389929334_cac86c8dd6e8dc183a40cedbd0c21b2d.jpg temp/1760775652_2781648_1389929307_94e5473a29a3224205f81518e077d018.jpg temp/1760775652_2781648_1389929301_9592c29531444811b92d1ffeb5ce66d0.jpg temp/1760775652_2781648_1389929278_72d2d6a76363947bd7a077d703726884.jpg temp/1760775652_2781648_1389929274_62b27c53ccfa8c0ec53c00059802fe65.jpg temp/1760775652_2781648_1389929253_51f2dd345e746e046e8d27cac0e31c1f.jpg temp/1760775652_2781648_1389929247_84808d0fa2ef3d5088d930117c0ab32c.jpg temp/1760775652_2781648_1389929241_741b9c05d9a1220cb0fb6434d1c20a19.jpg temp/1760775652_2781648_1389929231_675809a6506b92e0f2b9cb9dd3b72bd2.jpg temp/1760775652_2781648_1389929221_07231ec3e57c9ceee66881533890f4c7.jpg temp/1760775652_2781648_1389929214_393c5ad78bb321734595e8e80e15109a.jpg temp/1760775652_2781648_1389929193_ef5b31343af3827f48a063d3178ff081.jpg temp/1760775652_2781648_1389929182_f8c2e79c1f4d00650c32f990cfaa49f2.jpg temp/1760775652_2781648_1389929154_e812bfae7d3d81b89e2c80b1c9e0f198.jpg temp/1760775652_2781648_1389929127_65440fcc7f6961baa98c6325d2c1a3f1.jpg temp/1760775652_2781648_1389929124_33bcd729d0e8450922cc91b80e353771.jpg temp/1760775652_2781648_1389929098_8ecdf95754c65016f625f23bcaf0e0d0.jpg temp/1760775652_2781648_1389929096_f7872a3c49821094511e06e1d28a9d9b.jpg temp/1760775652_2781648_1389929087_f74a5a00042a1b18bd40134f45a32751.jpg temp/1760775652_2781648_1389929066_43449a9183fd39d2c8c40b062b12c119.jpg temp/1760775652_2781648_1389929064_b08e2e2137495e1a4f70f4236a1b0fec.jpg temp/1760775652_2781648_1389928970_c365648924276e94c3726a96bcf710d3.jpg temp/1760775652_2781648_1389928965_20f0aa81fbb734d1f6d3ae9dcbef3bd9.jpg temp/1760775652_2781648_1389928937_66bbb2b73a34b751f3ae9873186f11b7.jpg temp/1760775652_2781648_1389928907_56e12d5402ec80f250898654525ca7b0.jpg temp/1760775652_2781648_1389928906_019a3b92a0a30408be3a42d15b488bbb.jpg temp/1760775652_2781648_1389928879_d084875c110f87982ce2f9ab8e9e4823.jpg temp/1760775652_2781648_1389928878_756592a1b04db926ca5bf8c46af00cc3.jpg temp/1760775652_2781648_1389928848_b89e2c26dd92c084d2d985ed859045f8.jpg temp/1760775652_2781648_1389928818_6536c4c7626c9db61287de81ae35486e.jpg temp/1760775652_2781648_1389928626_22f4c2bc7a228dcb088c5b15ed122dd1.jpg temp/1760775652_2781648_1389928616_1e5cd5eb78cd3de491fe214435731fad.jpg temp/1760775652_2781648_1389928588_15a26f89d41244b7a2eb16de4a27b7c9.jpg temp/1760775652_2781648_1389928582_e1ec92d3efdb69e98180c700e117245e.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2339, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2533, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1389931725, 1389931705, 1389931662, 1389931636, 1389931632, 1389931570, 1389931562, 1389931557, 1389931498, 1389931495, 1389931492, 1389931490, 1389931432, 1389931431, 1389931426, 1389931420, 1389931390, 1389931377, 1389931372, 1389931350, 1389931343, 1389931342, 1389931340, 1389931311, 1389931302, 1389931273, 1389931262, 1389931259, 1389931255, 1389931239, 1389931212, 1389931166, 1389931149, 1389931137, 1389931104, 1389931082, 1389931074, 1389931056, 1389931049, 1389931033, 1389931019, 1389930996, 1389930986, 1389930982, 1389930981, 1389930956, 1389930927, 1389930925, 1389930920, 1389930897, 1389930862, 1389930861, 1389930860, 1389930817, 1389930806, 1389930796, 1389930795, 1389930788, 1389930747, 1389930745, 1389930736, 1389930732, 1389930721, 1389930715, 1389930683, 1389930674, 1389930658, 1389930657, 1389930644, 1389930620, 1389930596, 1389930590, 1389930589, 1389930579, 1389930555, 1389930540, 1389930534, 1389930528, 1389930519, 1389930514, 1389930493, 1389930487, 1389930472, 1389930463, 1389930443, 1389930434, 1389930419, 1389930403, 1389930398, 1389930390, 1389930381, 1389930363, 1389930349, 1389930336, 1389930331, 1389930284, 1389930273, 1389930225, 1389930215, 1389930213, 1389930182, 1389930175, 1389930169, 1389930161, 1389930137, 1389930130, 1389930129, 1389930126, 1389930117, 1389930114, 1389930102, 1389930101, 1389930095, 1389930094, 1389930078, 1389930069, 1389930068, 1389930062, 1389930061, 1389930051, 1389930050, 1389930046, 1389930034, 1389930028, 1389930019, 1389930017, 1389930014, 1389930013, 1389929995, 1389929986, 1389929979, 1389929975, 1389929969, 1389929950, 1389929935, 1389929910, 1389929899, 1389929879, 1389929831, 1389929820, 1389929808, 1389929787, 1389929786, 1389929737, 1389929714, 1389929693, 1389929683, 1389929663, 1389929660, 1389929654, 1389929630, 1389929619, 1389929618, 1389929601, 1389929584, 1389929583, 1389929568, 1389929567, 1389929554, 1389929546, 1389929540, 1389929538, 1389929534, 1389929518, 1389929494, 1389929466, 1389929334, 1389929307, 1389929301, 1389929278, 1389929274, 1389929253, 1389929247, 1389929241, 1389929231, 1389929221, 1389929214, 1389929193, 1389929182, 1389929154, 1389929127, 1389929124, 1389929098, 1389929096, 1389929087, 1389929066, 1389929064, 1389928970, 1389928965, 1389928937, 1389928907, 1389928906, 1389928879, 1389928878, 1389928848, 1389928818, 1389928626, 1389928616, 1389928588, 1389928582] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 47.33149766921997 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 13.89681363105774 time spend to save output : 47.33754849433899 total time spend for step 0 : 61.23436212539673 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.25user 4.10system 2:03.75elapsed 14%CPU (0avgtext+0avgdata 1887740maxresident)k 59848inputs+640152outputs (294major+694743minor)pagefaults 0swaps