python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 651155 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Date 2026-05-27 14:40:29.095081 Current got : datou_id : 533, datou_cur_ids : ['4383935'] with mtr_portfolio_ids : ['31123154'] and first list_photo_ids : [] new path : /proc/651155/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 131 ; length of list_pids : 131 ; length of list_args : 131 time to download the photos : 31.79436731338501 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Wed May 27 14:41:00 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2026-05-27 14:41:09.033210: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2026-05-27 14:41:09.062828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-05-27 14:41:09.063104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-05-27 14:41:09.065457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-05-27 14:41:09.067651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-05-27 14:41:09.068011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-05-27 14:41:09.070097: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-05-27 14:41:09.071309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-05-27 14:41:09.075709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-05-27 14:41:09.076744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-05-27 14:41:09.077182: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2026-05-27 14:41:09.102610: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2026-05-27 14:41:09.104674: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8b48000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2026-05-27 14:41:09.104704: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2026-05-27 14:41:09.235695: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d6e6740 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2026-05-27 14:41:09.235740: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2026-05-27 14:41:09.236599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-05-27 14:41:09.236695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-05-27 14:41:09.236725: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-05-27 14:41:09.236752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-05-27 14:41:09.236778: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-05-27 14:41:09.236805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-05-27 14:41:09.236830: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-05-27 14:41:09.236857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-05-27 14:41:09.238185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-05-27 14:41:09.238253: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-05-27 14:41:09.239072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-05-27 14:41:09.239100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-05-27 14:41:09.239114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-05-27 14:41:09.240529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2625 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2026-05-27 14:41:16.050235: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-05-27 14:41:16.050881: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-05-27 14:41:16.051498: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:14 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:27 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:28 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2026-03-04 13:32:04 create time in s3 : 2026-03-04 12:24:36 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.789516925811768 time used to load_weights : 0.41115498542785645 0it [00:00, ?it/s] 7it [00:00, 45.48it/s] 13it [00:00, 51.75it/s] 19it [00:00, 51.08it/s] 25it [00:00, 45.62it/s] 31it [00:00, 48.29it/s] 36it [00:00, 43.44it/s] 42it [00:00, 45.78it/s] 47it [00:01, 44.47it/s] 53it [00:01, 44.91it/s] 58it [00:01, 39.68it/s] 63it [00:01, 39.64it/s] 68it [00:01, 38.43it/s] 73it [00:01, 41.04it/s] 78it [00:01, 41.08it/s] 83it [00:01, 39.71it/s] 88it [00:02, 38.75it/s] 92it [00:02, 38.64it/s] 96it [00:02, 37.21it/s] 100it [00:02, 36.00it/s] 104it [00:02, 34.75it/s] 108it [00:02, 30.70it/s] 112it [00:02, 26.83it/s] 115it [00:03, 23.28it/s]temp/1779885629_651155_1417838323_26e9a7f4392364a572a0bba89bec57c6.jpg temp/1779885629_651155_1417838322_9dcc7ba209db8d00500645c49b0e1b86.jpg temp/1779885629_651155_1417838321_9416650a5bdbc9641ab8799bad320ec5.jpg temp/1779885629_651155_1417838320_0a24016a5359024aeb65d1ba0f6db46d.jpg temp/1779885629_651155_1417838319_91e13a50d4f70cb17f31fbbab7619ebf.jpg temp/1779885629_651155_1417838318_cd18e2c55fc83bf440cfe71289112004.jpg temp/1779885629_651155_1417838303_2ec0c7d2d49d04c742d506e2ebf589bc.jpg temp/1779885629_651155_1417838302_4b22ab8f35a7145157ce809b3aedf0bd.jpg temp/1779885629_651155_1417838301_e1cee78e794b7d7bfd1f028ab57c748f.jpg temp/1779885629_651155_1417838300_137f186d01f574db98b66dd0407c45d5.jpg temp/1779885629_651155_1417838299_fa0fc722ebddf083cd7e421d7fab90c1.jpg temp/1779885629_651155_1417838298_6d0d292cd4f02204f1de03c584cab945.jpg temp/1779885629_651155_1417838297_8c978be3ded1c5db1d2f57783510745c.jpg temp/1779885629_651155_1417838296_72349e3f5b0293e08ae27658d5057a58.jpg temp/1779885629_651155_1417838295_d78021eaa04fdba7dd4b9b9242d28319.jpg temp/1779885629_651155_1417838294_0a27f9ce5bf7fbb2aa640f7f65eb2e78.jpg temp/1779885629_651155_1417838293_82860eec6be0a0edb2ba79c649a3ff52.jpg temp/1779885629_651155_1417838292_2ae69316e4a6409d3a39cb8c89ecbc02.jpg temp/1779885629_651155_1417838283_6ec5d04ecfd120613d0d385d8d8092f3.jpg temp/1779885629_651155_1417838282_090ee046a1be1397ef627c2ae98ca87f.jpg temp/1779885629_651155_1417838281_ee8ab8bb0847591eaa2469c4b1528949.jpg temp/1779885629_651155_1417838280_babf55ea7ef251e4d2502a19c0236cd6.jpg temp/1779885629_651155_1417838279_89ed505495dc55ad6251ebb802754a94.jpg temp/1779885629_651155_1417838278_3b4fd5e9ed189e5a50dc121c6ababb37.jpg temp/1779885629_651155_1417838277_a07bc92954149580d29b821ee7883469.jpg temp/1779885629_651155_1417838275_45129105a98ba40587f31d35e5ad2c5b.jpg temp/1779885629_651155_1417838274_92d39f4a88ff4916d09f5f01e53f70c1.jpg temp/1779885629_651155_1417838273_02de963ccceb3664ba26e7062f3d63e9.jpg temp/1779885629_651155_1417838271_9343866c029079e95b3de07da0efc30c.jpg temp/1779885629_651155_1417838270_d69c8f5bb16164cd8db682cc7de804c7.jpg temp/1779885629_651155_1417838242_c251f260be015f76e90a4ce31db0fda8.jpg temp/1779885629_651155_1417838241_03ab1b16aa89f3d5452acdb76103333b.jpg temp/1779885629_651155_1417838240_e726d0383cf6a59d29e89a5dac682e5d.jpg temp/1779885629_651155_1417838239_9b5f8e634a48739543db917451ed5a06.jpg temp/1779885629_651155_1417838237_22703c03fe1b16c93ec00962b3e59b10.jpg temp/1779885629_651155_1417838236_77233a8ff6d2e5315c71e8c14286b23a.jpg temp/1779885629_651155_1417838216_0e905a636d00a0cdfabc2639c5bd5cb8.jpg temp/1779885629_651155_1417838215_04cf713afd3e75cc9963f92729f07107.jpg temp/1779885629_651155_1417838214_78d00473200d9bd142a6684033680689.jpg temp/1779885629_651155_1417838212_d4f90d1f48397322e58387f2189dee9d.jpg temp/1779885629_651155_1417838211_b663b467dc299113f971148298a71ec8.jpg temp/1779885629_651155_1417838210_d965bd6d6a8a372ab55113ffb37334e9.jpg temp/1779885629_651155_1417838199_4381110f4aa1541c2abdde860cf4e04c.jpg temp/1779885629_651155_1417838198_7826134714f0222d50395a43de0c92dc.jpg temp/1779885629_651155_1417838197_c4b02a2bd7ab0671d1b7a2596066bb04.jpg temp/1779885629_651155_1417838196_5946a3cdd9bbc277ee720ec6a2e6606a.jpg temp/1779885629_651155_1417838195_272acbc1b9ee186aa2dc5384255d0ea5.jpg temp/1779885629_651155_1417838194_43a5085d2ff2d4960fd163b9eb96493b.jpg temp/1779885629_651155_1417838186_0738b9d0424b46ed2f41dd466ab98f48.jpg temp/1779885629_651155_1417838185_c77a776bddaa506853f859c71171ef8e.jpg temp/1779885629_651155_1417838184_2d6d41b58b8c6728dbb80aea516fc6db.jpg temp/1779885629_651155_1417838183_7e6650b788e351c0bb68cc6a734b113a.jpg temp/1779885629_651155_1417838182_5775208bdb6dbe1cbca1b9405bbc49b3.jpg temp/1779885629_651155_1417838181_a99236e306d27ef3d34e21b755564d28.jpg temp/1779885629_651155_1417838148_c301b7451b814522f58fe29eed1f7930.jpg temp/1779885629_651155_1417838147_c673aa20f8c9a47191e466e3fe616ea3.jpg temp/1779885629_651155_1417838146_1d739482b9519538f122162b953b21f7.jpg temp/1779885629_651155_1417838145_595c7a0e42efe1986485e066e1d7882b.jpg temp/1779885629_651155_1417838144_b1a81f59b43fa7c9fefce5f6c7286520.jpg temp/1779885629_651155_1417838143_7f6a3e0dcb293418e4ad87dcb9607ee9.jpg temp/1779885629_651155_1417838125_291d4499f7a7acaaedcb1a4450ed49e5.jpg temp/1779885629_651155_1417838124_bb4443c43371ceea51ae5bc3549b7c8a.jpg temp/1779885629_651155_1417838123_04bf00e4e4cd5b02c73c1e6d9a806307.jpg temp/1779885629_651155_1417838122_b28923fbc73fdc0758a5483735f06c87.jpg temp/1779885629_651155_1417838121_79b200cfcfcc631a7d8f66a379f5373e.jpg temp/1779885629_651155_1417838120_5ca74e584909471719c3913ea82b80f5.jpg temp/1779885629_651155_1417838112_aabca9534bed947b91aa73ede02359b2.jpg temp/1779885629_651155_1417838110_a7500ba3e9d0586a34a0ebada38e5b37.jpg temp/1779885629_651155_1417838109_5843b78e1063ddd6f74dac6531b5c7e1.jpg temp/1779885629_651155_1417838108_31ed759e743c52de04cd13dc165c53a3.jpg temp/1779885629_651155_1417838107_186ed395b3148b44a536b4dbe3e2dbd5.jpg temp/1779885629_651155_1417838106_2f171bd90bd353a9c7b6aa5a2c67ba89.jpg temp/1779885629_651155_1417838105_53bde4e3b51395ea7536ddfd52cea9ac.jpg temp/1779885629_651155_1417838104_53a8e657bc613ef15290a28061cbc901.jpg temp/1779885629_651155_1417838103_5f16143d97886fdd40c1a1f3a6fddcf3.jpg temp/1779885629_651155_1417838102_5301a062bf6266b7e6f31d8632309501.jpg temp/1779885629_651155_1417838101_f505e0721863ee6ae4a3cf2989cb8d54.jpg temp/1779885629_651155_1417838100_da666d5393cf62aed3dce3568fd5cc27.jpg temp/1779885629_651155_1417838099_a63bf4a9ea88936b214bd7b8c54ee073.jpg temp/1779885629_651155_1417838098_6199c8f8beb7179e66e017f08e9620d6.jpg temp/1779885629_651155_1417838097_509681bf449c155f20bc5133d5bd4b09.jpg temp/1779885629_651155_1417838096_212fb23f6531ed20897de2ed3dd4d2aa.jpg temp/1779885629_651155_1417838095_c47a6f0b69e2b457a31f495beff26aab.jpg temp/1779885629_651155_1417838092_48a2dc76f22db477558696fa7c989310.jpg temp/1779885629_651155_1417838091_6657e77d2da5c3c5debf133848f2243c.jpg temp/1779885629_651155_1417838090_a8ba235d287527977a2c740ddc7bccb6.jpg temp/1779885629_651155_1417838089_c6bb3071ed2ac9a1b8cb29518c71f407.jpg temp/1779885629_651155_1417838088_8195b16f0054f6343f2c2363746a38dd.jpg temp/1779885629_651155_1417838087_abbdc418ae43544ec361ece5835f1a7b.jpg temp/1779885629_651155_1417838078_2d5aca6c298e0cb68c4f223f8bbcdb8d.jpg temp/1779885629_651155_1417838077_f1e2952b8e1d671ba9675d36576173ed.jpg temp/1779885629_651155_1417838076_d703138d01516a8e011b37058ccd4dea.jpg temp/1779885629_651155_1417838075_3f2153e65694235e1f6038be0e236817.jpg temp/1779885629_651155_1417838074_c8fcb0e599d1b2486862a31b0565cea8.jpg temp/1779885629_651155_1417838073_ef9879d24f2dafa9d1fc6e01b0b3474a.jpg temp/1779885629_651155_1417838064_e180c24f0871bcd4d6040a56bec2b677.jpg temp/1779885629_651155_1417838063_07a0cc22532b32bc629843b7ecaf40c2.jpg temp/1779885629_651155_1417838062_e4a72b574ce9101b8a67d9def430c0de.jpg temp/1779885629_651155_1417838061_2c9f972646e5781c33765693022c95b1.jpg temp/1779885629_651155_1417838060_1b30664d216d257b29d982944a4c1a16.jpg temp/1779885629_651155_1417838059_110d663ba4468a98369db8bc35388bee.jpg temp/1779885629_651155_1417838058_27b06cd0aa23c57125b4f7bfc4eb1680.jpg temp/1779885629_651155_1417838057_08c6c35dc6fa06812dadc8a408362b58.jpg temp/1779885629_651155_1417838056_90a97fe91f32ce515b39cdbdccaffe5f.jpg temp/1779885629_651155_1417838055_c56017e9379dc2c2967d84b1d59518ee.jpg temp/1779885629_651155_1417838054_72765bd9f3ecd06b0f36cc3689fb39b5.jpg temp/1779885629_651155_1417838053_a0dbc6bb52f6b0307a5b40bb953028b0.jpg temp/1779885629_651155_1417838044_98b241c6affb833e1c0d53e5f739fb42.jpg temp/1779885629_651155_1417838043_98e5a3f6f34f6727432233471b3fd59e.jpg temp/1779885629_651155_1417838042_5b2ae824980db5b3062cc2f9d11a4ddd.jpg temp/1779885629_651155_1417838041_d65923aeeb3e2faaf02d445c31d82065.jpg temp/1779885629_651155_1417838040_338360a15da3404f5bd2f01702c4bbb1.jpg temp/1779885629_651155_1417838039_184bc0cbea8869e8c103eecb8efe8ece.jpg temp/1779885629_651155_1417838038_29e40a5ef30bec5025e16a9b7c189309.jpg temp/1779885629_651155_1417838037_d868ed359032b04c4b7a4591fd7401bb.jpg 118it [00:03, 20.84it/s] 121it [00:03, 19.78it/s] 124it [00:03, 21.26it/s] 127it [00:03, 19.78it/s] 130it [00:03, 18.52it/s] 131it [00:03, 32.86it/s]2026-05-27 14:41:23.889526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-05-27 14:41:24.465570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-05-27 14:41:24.499202: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2026-05-27 14:41:24.508266: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1779885629_651155_1417838036_7ea2948340f060da00bfd80fa5fa3658.jpg temp/1779885629_651155_1417838035_28b3148f2f9411c0172bbab8f4f67bde.jpg temp/1779885629_651155_1417838034_54e6fe7d3bf9b3547eada1bb1e4f81dc.jpg temp/1779885629_651155_1417838032_03c9ff19c85b785e77760b59e2dd38d1.jpg temp/1779885629_651155_1417838017_97ed36d83a3c2127f2a31d7973cf2792.jpg temp/1779885629_651155_1417838016_cc547b187c8cb69970a74d233451e4fb.jpg temp/1779885629_651155_1417838015_977b4fe84cd4672e75422d21da3f4497.jpg temp/1779885629_651155_1417838014_2d587428432ee63f5fe768322f0c708c.jpg temp/1779885629_651155_1417838013_df1c4d8d3335ea112b569ab17bc9a61b.jpg temp/1779885629_651155_1417838012_73b0127c4236e912f1ca848102c0e098.jpg temp/1779885629_651155_1417838005_62ea7a5fc00783b1e9f28295bc43d942.jpg temp/1779885629_651155_1417838004_b1e620f89bc6dda2648ec5e24443da47.jpg temp/1779885629_651155_1417838003_85b85c590d01ea61f3ae1e44c018e356.jpg temp/1779885629_651155_1417838002_bbdce7a65ff8839c3af6ebbd7a6b7c1a.jpg temp/1779885629_651155_1417838001_334bf03f954a0fda18c67f358e1c3642.jpg temp/1779885629_651155_1417838000_07baf18429d2160728a9694e61b579bf.jpg Found 131 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2340, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2534, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3171, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1417838323, 1417838322, 1417838321, 1417838320, 1417838319, 1417838318, 1417838303, 1417838302, 1417838301, 1417838300, 1417838299, 1417838298, 1417838297, 1417838296, 1417838295, 1417838294, 1417838293, 1417838292, 1417838283, 1417838282, 1417838281, 1417838280, 1417838279, 1417838278, 1417838277, 1417838275, 1417838274, 1417838273, 1417838271, 1417838270, 1417838242, 1417838241, 1417838240, 1417838239, 1417838237, 1417838236, 1417838216, 1417838215, 1417838214, 1417838212, 1417838211, 1417838210, 1417838199, 1417838198, 1417838197, 1417838196, 1417838195, 1417838194, 1417838186, 1417838185, 1417838184, 1417838183, 1417838182, 1417838181, 1417838148, 1417838147, 1417838146, 1417838145, 1417838144, 1417838143, 1417838125, 1417838124, 1417838123, 1417838122, 1417838121, 1417838120, 1417838112, 1417838110, 1417838109, 1417838108, 1417838107, 1417838106, 1417838105, 1417838104, 1417838103, 1417838102, 1417838101, 1417838100, 1417838099, 1417838098, 1417838097, 1417838096, 1417838095, 1417838092, 1417838091, 1417838090, 1417838089, 1417838088, 1417838087, 1417838078, 1417838077, 1417838076, 1417838075, 1417838074, 1417838073, 1417838064, 1417838063, 1417838062, 1417838061, 1417838060, 1417838059, 1417838058, 1417838057, 1417838056, 1417838055, 1417838054, 1417838053, 1417838044, 1417838043, 1417838042, 1417838041, 1417838040, 1417838039, 1417838038, 1417838037, 1417838036, 1417838035, 1417838034, 1417838032, 1417838017, 1417838016, 1417838015, 1417838014, 1417838013, 1417838012, 1417838005, 1417838004, 1417838003, 1417838002, 1417838001, 1417838000] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 131 time used for this insertion : 0.05283856391906738 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 23.532172441482544 time spend to save output : 0.06045651435852051 total time spend for step 0 : 23.592628955841064 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.54user 8.90system 1:00.50elapsed 38%CPU (0avgtext+0avgdata 1979732maxresident)k 2190312inputs+3818256outputs (682major+734748minor)pagefaults 0swaps