python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1035314 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2961>200 please execute split_portfolio.py -i 25245842 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3331981'] with mtr_portfolio_ids : ['25245842'] and first list_photo_ids : [] new path : /proc/1035314/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 114 ; length of list_pids : 114 ; length of list_args : 114 time to download the photos : 18.609408617019653 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu Jul 24 10:20:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-07-24 10:20:51.266944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-07-24 10:20:51.300483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-24 10:20:51.300743: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-24 10:20:51.302951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-24 10:20:51.304766: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-24 10:20:51.305108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-24 10:20:51.307038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-24 10:20:51.308221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-24 10:20:51.312314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-24 10:20:51.313430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-24 10:20:51.313777: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-07-24 10:20:51.343423: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493035000 Hz 2025-07-24 10:20:51.345393: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f3074000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-07-24 10:20:51.345445: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-07-24 10:20:51.483724: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x33a20e00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-07-24 10:20:51.483818: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-07-24 10:20:51.484970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-24 10:20:51.485131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-24 10:20:51.485180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-24 10:20:51.485224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-24 10:20:51.485263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-24 10:20:51.485299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-24 10:20:51.485341: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-24 10:20:51.485384: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-24 10:20:51.486711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-24 10:20:51.486814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-24 10:20:51.487681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-07-24 10:20:51.487711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-07-24 10:20:51.487723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-07-24 10:20:51.488996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3231 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-07-24 10:20:59.517381: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-24 10:20:59.518234: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 10.273552179336548 time used to load_weights : 0.744713306427002 0it [00:00, ?it/s] 42it [00:00, 414.27it/s] 94it [00:00, 472.23it/s]temp/1753345229_1035314_1373502624_9b9aee387a61eeead50ea6aa85a983d7.jpg temp/1753345229_1035314_1373502622_0044ccb3b1d7a0b09968fd79b6b29fdf.jpg temp/1753345229_1035314_1373502620_82a242f9bc24aa55d527b105d005374f.jpg temp/1753345229_1035314_1373502616_ca175188da338a1963db9c4f690f9386.jpg temp/1753345229_1035314_1373502612_a85f148d1b2e7aab47c7ae539380addc.jpg temp/1753345229_1035314_1373502594_99eb9125a474658b7d1463c69b9d20e5.jpg temp/1753345229_1035314_1373502577_fca1e4f12992af834054fe65a1fb8025.jpg temp/1753345229_1035314_1373502576_0ea4f6d98f1d8c6084ae6711a23a082b.jpg temp/1753345229_1035314_1373502573_bdbc8dd53d13a20e096208eec5113d7a.jpg temp/1753345229_1035314_1373502571_862c04bafaf6ad5e8db326ca6903a9a2.jpg temp/1753345229_1035314_1373502566_f76df2f669239bc38a32fda0cd3e4328.jpg temp/1753345229_1035314_1373502563_a1da8c72bb41ecd2d9b7421979791f9b.jpg temp/1753345229_1035314_1373502542_f187d38303136c03255cd7755f618e79.jpg temp/1753345229_1035314_1373502540_2947a040242a080ec7fd3f200024096f.jpg temp/1753345229_1035314_1373502537_e96a3fca3a0109e6177da10e232b4b46.jpg temp/1753345229_1035314_1373502535_0205ea2039262081c00b9fdb5e2583e0.jpg temp/1753345229_1035314_1373502529_2c96edc9f4861db5d4765f03df9a5f07.jpg temp/1753345229_1035314_1373502507_4c9297e32d9835531f2e5c1d0aebb5b9.jpg temp/1753345229_1035314_1373502502_16d702b4af4da52deda1d8dd58b0e50a.jpg temp/1753345229_1035314_1373502498_f3744eb2081da4e98a89225cd7448292.jpg temp/1753345229_1035314_1373502496_45e28b55c7ed0462e5e856967fcfc305.jpg temp/1753345229_1035314_1373502493_776ea89bd032a699f009ef840200f176.jpg temp/1753345229_1035314_1373502490_e3da17792a78ef9cdef37c04956878c4.jpg temp/1753345229_1035314_1373498236_34f78818839c6d29ee6e1122ae0c59fb.jpg temp/1753345229_1035314_1373498234_d40d05c60798097d45e43e59daa1128f.jpg temp/1753345229_1035314_1373498231_424d7b9a10dd9c4fd4ca1fed3ad2365c.jpg temp/1753345229_1035314_1373498228_ba5be6f0896709f92be271ab351395e2.jpg temp/1753345229_1035314_1373498225_3969bee879af5a18d5bd3260e9ee96ea.jpg temp/1753345229_1035314_1373498220_9728e7a370863281c95066877f74f3dd.jpg temp/1753345229_1035314_1373498184_b3e902e0cef68514e8d74bbda42337cb.jpg temp/1753345229_1035314_1373498169_19625cae6bfa7562c33f268fd1716a62.jpg temp/1753345229_1035314_1373498137_061a9b388830b3287e6af22c71418e1d.jpg temp/1753345229_1035314_1373498135_12da717d7cbdd5c5125b3465e49dee03.jpg temp/1753345229_1035314_1373498133_62d29b095e9d3258f5eace4ae0304389.jpg temp/1753345229_1035314_1373498130_4a4216d6843a20ecd7804475ba2b8e14.jpg temp/1753345229_1035314_1373498107_00eeb2fa66498461e18afbbf4d6f69e4.jpg temp/1753345229_1035314_1373498105_87599023ebb9b4ccd697e76ef3472c35.jpg temp/1753345229_1035314_1373498103_29425858a7a9060f1df1311fc2bce9cb.jpg temp/1753345229_1035314_1373498099_fae6cc2013b77c69f532669e9c899921.jpg temp/1753345229_1035314_1373498084_e0a440fa10987fe6a894ec3d5e44bd97.jpg temp/1753345229_1035314_1373498054_a2754396f853b9fb297046db8c15bc65.jpg temp/1753345229_1035314_1373497940_a4f30acff4d3665a4ea461801971f997.jpg temp/1753345229_1035314_1373497937_dfc10002994e22617fe946e1a98e2dc0.jpg temp/1753345229_1035314_1373497934_314b9dd4de70167391e56edfdf89c1ce.jpg temp/1753345229_1035314_1373497931_8fbfcc568db11d3df28c679e78e9d921.jpg temp/1753345229_1035314_1373497927_11d28ad34b1667341e9a7a8a7f57542a.jpg temp/1753345229_1035314_1373497925_0baffd3b604f13861d3b7b4c1558f55f.jpg temp/1753345229_1035314_1373497899_b41f9bc364d3e28f81ec1553057b2af6.jpg temp/1753345229_1035314_1373497897_4f2896ce56c65f51a058414487000e82.jpg temp/1753345229_1035314_1373497893_d1c3710664f5d651faa79cd450fc58e5.jpg temp/1753345229_1035314_1373497889_36827f46b262e19d7557fb7d26f74e78.jpg temp/1753345229_1035314_1373497886_cd1a4742eeaf1293763d19e833e9fc7c.jpg temp/1753345229_1035314_1373497883_fae29a8a9a504ae2efd805bcf30c8ccf.jpg temp/1753345229_1035314_1373497868_d7756c6d12f2b957d55888e485717056.jpg temp/1753345229_1035314_1373497866_7b95f5bec13edff64293a7b7ca47b72f.jpg temp/1753345229_1035314_1373497864_0a90d1982bb0887ed68ab1ccce870645.jpg temp/1753345229_1035314_1373497863_3fce629cbda9e041aaab9204bb6fb04c.jpg temp/1753345229_1035314_1373497860_8d26f3adeb584949589c3a920ad28711.jpg temp/1753345229_1035314_1373497858_3b32f163fc7bc85164cd40f095bd79a0.jpg temp/1753345229_1035314_1373497843_e9194660d4e9d72aacbda7a44a09760a.jpg temp/1753345229_1035314_1373497842_49fe354a4be95ce99dda9d3d1a538898.jpg temp/1753345229_1035314_1373497839_47da6db0ba52f4d40ccdb0a7689245d6.jpg temp/1753345229_1035314_1373497836_47990a25168dc1a549f6717b26660bb0.jpg temp/1753345229_1035314_1373497825_e84ca81202477fad21d701c7d08c0c31.jpg temp/1753345229_1035314_1373497810_11e83461c2ac6731f3c8b9bb51efe5fb.jpg temp/1753345229_1035314_1373497758_d514e0b64952876c77380025202b4a7f.jpg temp/1753345229_1035314_1373497755_168b3f17cea31da140ba4f327cb9e39d.jpg temp/1753345229_1035314_1373497749_431577ae1da97a073225712fdff2a2ef.jpg temp/1753345229_1035314_1373497746_468e16d837701dc6f7ab94b865f7ec24.jpg temp/1753345229_1035314_1373497742_0b5cf367065f8249e1c3a6efcd752b0d.jpg temp/1753345229_1035314_1373497732_ca49b6d2d07fd5e262f4aa12c5dad411.jpg temp/1753345229_1035314_1373497712_608a03c2c6418205a58881aae002677c.jpg temp/1753345229_1035314_1373497709_9c6c21f979656f34f660a2d8533432ce.jpg temp/1753345229_1035314_1373497707_0046de6ac95ce6f1bc2ed69c3cbfadbc.jpg temp/1753345229_1035314_1373497704_2cbddf3a331558571040550db24cc58c.jpg temp/1753345229_1035314_1373497702_a02985a16e6230110481c280b2e6f00c.jpg temp/1753345229_1035314_1373497700_16f7ca88aa32712af6cebf6b1060bf61.jpg temp/1753345229_1035314_1373497690_90f865b7575c8fbc4fa010a828cdbf85.jpg temp/1753345229_1035314_1373497686_667ed33e48f3677b1f16cb857a19ab53.jpg temp/1753345229_1035314_1373497673_2e90b9152a92bf7fd2095525e5c74988.jpg temp/1753345229_1035314_1373497662_bc65eefd57f7781cb6a45111cf0d5deb.jpg temp/1753345229_1035314_1373497658_38aa1eecc98ad83214c229033294c5a0.jpg temp/1753345229_1035314_1373497654_3b83bf3d5d1495dccaf31da4bfb270d8.jpg temp/1753345229_1035314_1373497652_6a0d19fae4f4b9bd0f31327a3930ef35.jpg temp/1753345229_1035314_1373497650_a4e875924acc61af1fecd8b8f66930fe.jpg temp/1753345229_1035314_1373497646_9494c3fc682d06abdc5c037183a59907.jpg temp/1753345229_1035314_1373497623_1cc4e8c624d21ace9a9c922380529eea.jpg temp/1753345229_1035314_1373497620_26344a869a791c919afeca4f036744fc.jpg temp/1753345229_1035314_1373497616_cfbb97a7025a3c54ce4a688ca55d6b10.jpg temp/1753345229_1035314_1373497614_ae5d589e7e9b3896397df5e445153c56.jpg temp/1753345229_1035314_1373497611_344b0b14efefefd657ed2c0d8acdb428.jpg temp/1753345229_1035314_1373497609_2ca03f512dabd23e29e7fead2c424251.jpg temp/1753345229_1035314_1373497593_36a3d775dd12faf56e4c3420ddceb11e.jpg temp/1753345229_1035314_1373497591_75721302433b5a041dcd53a63f1dedf3.jpg temp/1753345229_1035314_1373497590_798be7c0a4e1267084f8b5362be7365d.jpg temp/1753345229_1035314_1373497588_7428c6895ecfd8249696b0b19c076050.jpg temp/1753345229_1035314_1373497586_b95ade6a0522b0de687022a964f48488.jpg temp/1753345229_1035314_1373497584_df8e7e54bbca3c21c02571ee6baf95fd.jpg temp/1753345229_1035314_1373497581_17168fce008114eeffecf73ec0e5bf42.jpg temp/1753345229_1035314_1373497571_e1c7d1876f071274f14992a957060a18.jpg temp/1753345229_1035314_1373497497_c239e00fe4b5f68a1cfa6441afdaa5d9.jpg temp/1753345229_1035314_1373497468_c32cf8ce1878144aeb918b0a2a9e6451.jpg temp/1753345229_1035314_1373497349_bef6c4b2b5ed8a378ae1ebac6850b6bc.jpg temp/1753345229_1035314_1373497320_5b3b8b6b86f05de941785c80a4ac0dec.jpg temp/1753345229_1035314_1373497293_d517ca04510cf6a899fdccf6957ab8fd.jpg temp/1753345229_1035314_1373497244_51565f8c71a986db9d4b78b761205bd9.jpg temp/1753345229_1035314_1373497198_903da4b089930a0ea6f0c3a6c2f9ab58.jpg temp/1753345229_1035314_1373497145_0c3ba723b1dfc312468b5c6bf9aba6ef.jpg temp/1753345229_1035314_1373497074_4f38c0820d505def61eb4a104e656f44.jpg temp/1753345229_1035314_1373497056_2b9b4454a3a81d8414e8162e4802c501.jpg temp/1753345229_1035314_1373497044_db5f983c45ed8ceb9b2b4c205ecfe151.jpg temp/1753345229_1035314_1373497041_82f1f11f195faf76ae164ad30cea173f.jpg temp/1753345229_1035314_1373497039_74aabc8a068cc9e96a84d4f3165497ef.jpg 114it [00:00, 477.75it/s]2025-07-24 10:21:05.121868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-24 10:21:05.211564: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-07-24 10:21:05.293176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-24 10:21:05.296970: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-07-24 10:21:05.299831: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1753345229_1035314_1373497033_317297217ac74ba931917b8d7cd1d250.jpg Found 114 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1373502624, 1373502622, 1373502620, 1373502616, 1373502612, 1373502594, 1373502577, 1373502576, 1373502573, 1373502571, 1373502566, 1373502563, 1373502542, 1373502540, 1373502537, 1373502535, 1373502529, 1373502507, 1373502502, 1373502498, 1373502496, 1373502493, 1373502490, 1373498236, 1373498234, 1373498231, 1373498228, 1373498225, 1373498220, 1373498184, 1373498169, 1373498137, 1373498135, 1373498133, 1373498130, 1373498107, 1373498105, 1373498103, 1373498099, 1373498084, 1373498054, 1373497940, 1373497937, 1373497934, 1373497931, 1373497927, 1373497925, 1373497899, 1373497897, 1373497893, 1373497889, 1373497886, 1373497883, 1373497868, 1373497866, 1373497864, 1373497863, 1373497860, 1373497858, 1373497843, 1373497842, 1373497839, 1373497836, 1373497825, 1373497810, 1373497758, 1373497755, 1373497749, 1373497746, 1373497742, 1373497732, 1373497712, 1373497709, 1373497707, 1373497704, 1373497702, 1373497700, 1373497690, 1373497686, 1373497673, 1373497662, 1373497658, 1373497654, 1373497652, 1373497650, 1373497646, 1373497623, 1373497620, 1373497616, 1373497614, 1373497611, 1373497609, 1373497593, 1373497591, 1373497590, 1373497588, 1373497586, 1373497584, 1373497581, 1373497571, 1373497497, 1373497468, 1373497349, 1373497320, 1373497293, 1373497244, 1373497198, 1373497145, 1373497074, 1373497056, 1373497044, 1373497041, 1373497039, 1373497033] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 114 time used for this insertion : 0.07079505920410156 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 17.182086944580078 time spend to save output : 0.07856059074401855 total time spend for step 0 : 17.260647535324097 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 17.36user 4.97system 0:43.73elapsed 51%CPU (0avgtext+0avgdata 1788620maxresident)k 405312inputs+319568outputs (291major+618951minor)pagefaults 0swaps