python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3941189 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2832>200 please execute split_portfolio.py -i 23084261 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2885849'] with mtr_portfolio_ids : ['23084261'] and first list_photo_ids : [] new path : /proc/3941189/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 32.309303760528564 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat May 17 09:41:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1827 wait 20 seconds inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-05-17 09:41:30.034751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-17 09:41:30.063587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-17 09:41:30.063886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 09:41:30.066040: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 09:41:30.068181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-17 09:41:30.068511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-17 09:41:30.094515: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-17 09:41:30.095821: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-17 09:41:30.101543: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 09:41:30.102790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-17 09:41:30.103372: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-17 09:41:30.131163: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-17 09:41:30.132862: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fdf4c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-17 09:41:30.132908: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-17 09:41:30.354098: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xff2d3c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-17 09:41:30.354180: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-17 09:41:30.355069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-17 09:41:30.355172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 09:41:30.355193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 09:41:30.355212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-17 09:41:30.355232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-17 09:41:30.355251: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-17 09:41:30.355270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-17 09:41:30.355290: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 09:41:30.356083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-17 09:41:30.356129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-17 09:41:30.356651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-17 09:41:30.356665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-17 09:41:30.356673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-17 09:41:30.357489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2944 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-17 09:41:37.449662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-17 09:41:37.450303: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.367029666900635 time used to load_weights : 0.5772092342376709 0it [00:00, ?it/s] 105it [00:00, 1045.83it/s]temp/1747467629_3941189_1359086060_2ebd8af32651cc6963a88693b09da526.jpg temp/1747467629_3941189_1359086058_a4395a9c3622a27629181aa3f5aee938.jpg temp/1747467629_3941189_1359086057_e7826fcff5bb2cb337ff0bce34bdab99.jpg temp/1747467629_3941189_1359086056_650fa86c52713884ecff43c1d14b1d1d.jpg temp/1747467629_3941189_1359086001_91e823ed27b5660afe4dd15bbc821811.jpg temp/1747467629_3941189_1359085997_eaaf9cc9e8f2459894df199757403879.jpg temp/1747467629_3941189_1359085969_03ce6aa15e14f0090a7d333249b9d314.jpg temp/1747467629_3941189_1359085965_83916a569283751926b6ddcffe4ebfa9.jpg temp/1747467629_3941189_1359085961_6f49a7c874ad74664a08f25ba8a3beef.jpg temp/1747467629_3941189_1359085956_346d21bdf74abddb0c88f06ee37d94af.jpg temp/1747467629_3941189_1359085955_452fdbda3261cf53a2624c1a66254ec4.jpg temp/1747467629_3941189_1359085950_e00466a4b956f6d6d558f792758ca575.jpg temp/1747467629_3941189_1359085881_3191075272bcc426e9b78556c53bc759.jpg temp/1747467629_3941189_1359085877_797046d74a362c22fe5abed865101056.jpg temp/1747467629_3941189_1359085873_663f6158a6abbd34e0cea0607c91e2e7.jpg temp/1747467629_3941189_1359085871_215650102bc2c836b55fb9b1201f996a.jpg temp/1747467629_3941189_1359085870_a068ebbbfef0ac4af55e319208e52073.jpg temp/1747467629_3941189_1359085868_927e0712d41d3e679bb16fa1756e9f3c.jpg temp/1747467629_3941189_1359085852_618808a26d7dcd2073cc621abe0262d0.jpg temp/1747467629_3941189_1359085851_ec58ff06c075fa6b15747c4d345801a3.jpg temp/1747467629_3941189_1359085850_8a0f47e4fd6cf8c5e8fd56dde3823f8a.jpg temp/1747467629_3941189_1359085849_29936eae3dd80f7cf9cef32b7e80e7e1.jpg temp/1747467629_3941189_1359085848_91719eba98e7efc4cc988dd09b32a0b0.jpg temp/1747467629_3941189_1359085846_85a589181fd39254ecf081e507dcd2b9.jpg temp/1747467629_3941189_1359085830_e9979b8ba847ff713f348870dcf8e26a.jpg temp/1747467629_3941189_1359085829_077e11a8b260a0d4e3b48c0823f8a97c.jpg temp/1747467629_3941189_1359085799_4468620383ef9170287e6dcc2a41ad17.jpg temp/1747467629_3941189_1359085798_ef02d1b24a8e13a27f9bda2ea42d710f.jpg temp/1747467629_3941189_1359085797_5f904e888cdcf916b9c224bbee672f61.jpg temp/1747467629_3941189_1359085796_ce63982ff34d9b6295b7ec6a3b9a4d67.jpg temp/1747467629_3941189_1359079921_4881d733124591850e2ec225cac39062.jpg temp/1747467629_3941189_1359079917_e6d0f936ba649a7a6bf76ee6a0651663.jpg temp/1747467629_3941189_1359079913_c64e59c9aa7243a175705fb53c3462a7.jpg temp/1747467629_3941189_1359079909_24d4ab91339139d876f472b04f00de8b.jpg temp/1747467629_3941189_1359079906_84b27eb6a472d67aa42b07a25bee7ae4.jpg temp/1747467629_3941189_1359079902_d53e418e12a55073093de4c658f30887.jpg temp/1747467629_3941189_1359079886_111669bcac88157f09c4696297b21296.jpg temp/1747467629_3941189_1359079882_1337243d7c9303b18d29dfd1180dfc1c.jpg temp/1747467629_3941189_1359079879_78f3dc5e1af555a6f5ca0e4af207d598.jpg temp/1747467629_3941189_1359079876_a4f8187b0484343d0cfd48cf2aa84d98.jpg temp/1747467629_3941189_1359079875_31a3348141d010b4b8c2df6943f23604.jpg temp/1747467629_3941189_1359079873_94eddb08d88ff4b8063f6d1cc8fceffb.jpg temp/1747467629_3941189_1359079858_9aebe93abf4200f807af57fc9e4543fb.jpg temp/1747467629_3941189_1359079855_f1daeaad9577076625e729381c8c6d1b.jpg temp/1747467629_3941189_1359079854_ca3897fb8b66e517b2d06dd9ed047d9a.jpg temp/1747467629_3941189_1359079850_977a30c850152a4c154583fbe5d4db68.jpg temp/1747467629_3941189_1359079848_3a33fd3590999cb462fe940afd175e59.jpg temp/1747467629_3941189_1359079845_630ac33436422005d70072879f75139b.jpg temp/1747467629_3941189_1359079830_6417a4ad00dd8e67a3c21cd745f1368c.jpg temp/1747467629_3941189_1359079828_2b826c11b0d5e669f53d2d1defb695e8.jpg temp/1747467629_3941189_1359079825_adc83b82ab8b9abc3ca80ee6d13db4c4.jpg temp/1747467629_3941189_1359079822_25250d919d385090222fb25a0a257c44.jpg temp/1747467629_3941189_1359079820_8113ab9428ae893f062dbab609961a6d.jpg temp/1747467629_3941189_1359079817_268ab917dd8171711ed95f268aeeee22.jpg temp/1747467629_3941189_1359079806_a45b226e171cb37f902d3a9e0a25a6d0.jpg temp/1747467629_3941189_1359079803_e2cb9f538ca8aadffbd4208be1ec49eb.jpg temp/1747467629_3941189_1359079800_3e77825071c4d25f56350fa0ba65d8cb.jpg temp/1747467629_3941189_1359079796_2e9e6086117a665c6b3ddca85a80bd86.jpg temp/1747467629_3941189_1359079794_d38ef30ab59899669df92589f6350262.jpg temp/1747467629_3941189_1359079791_e30dfda034ea5019aab10609eaa235e5.jpg temp/1747467629_3941189_1359079775_abadc4d6e5f626b6c8921705798eae1f.jpg temp/1747467629_3941189_1359079772_1508e65004931d166c0b6492da86e238.jpg temp/1747467629_3941189_1359079769_cbc71faee103954224d8d3a19f74bb81.jpg temp/1747467629_3941189_1359079766_c139c5990a7f7f9f81b4ad1d64ea3d91.jpg temp/1747467629_3941189_1359079765_c21fa08938c20789db168c749de1e9eb.jpg temp/1747467629_3941189_1359079763_72e7fc2d94cea5a2c7761fcf7e763452.jpg temp/1747467629_3941189_1359079736_dc3fc4caf16e723d4d0d45e77a413092.jpg temp/1747467629_3941189_1359079734_e2a9ce4776f406a779a8893043c3febf.jpg temp/1747467629_3941189_1359079733_4d498079f3156f623a3df0ea53b84984.jpg temp/1747467629_3941189_1359079731_427bf159164991dabbce1844a51a64f1.jpg temp/1747467629_3941189_1359079730_b9b69b5ea85e1abda39949e5793f4088.jpg temp/1747467629_3941189_1359079728_c70937cf92e9c445173b43208f662d4e.jpg temp/1747467629_3941189_1359079713_d1f864c9a4c749b3c4d0378c16c3e52b.jpg temp/1747467629_3941189_1359079712_fbf771d10c59020db094fd7e5f0ee10f.jpg temp/1747467629_3941189_1359079710_ef70a6145a6a995d484d31fa22cadfec.jpg temp/1747467629_3941189_1359079708_d060ca2dc7419c49eb71b8eecf6114f0.jpg temp/1747467629_3941189_1359079706_fa9e961f365917dae47ab84f43838b6f.jpg temp/1747467629_3941189_1359079702_91935a9d47ba1f104ca9273dbb37ca40.jpg temp/1747467629_3941189_1359079686_d0997ae8674a7b89708f05dc12c8bfe2.jpg temp/1747467629_3941189_1359079684_835e2f29e407be168eb909a2250737e8.jpg temp/1747467629_3941189_1359079682_b73c2ce84deb1c880bef3fceeae40d5c.jpg temp/1747467629_3941189_1359079679_92e63200d80d888a15504e499ea545ea.jpg temp/1747467629_3941189_1359079678_3357459e5410309bdd0cea7d65fa26c9.jpg temp/1747467629_3941189_1359079677_5224a7a0ec61456330773b0412cc86c2.jpg temp/1747467629_3941189_1359079452_a58e4d33174908ce1f82303f95ce5200.jpg temp/1747467629_3941189_1359079450_49e71c103d226cc5e07a11e166ead20c.jpg temp/1747467629_3941189_1359079448_7a8ec138becb0c478931b2b79dc1d6c0.jpg temp/1747467629_3941189_1359079446_9262b75828fbd730877b96a2bbbdc68a.jpg temp/1747467629_3941189_1359079445_d2a75e911b4869d42de3cbf145743797.jpg temp/1747467629_3941189_1359079443_f44e599b9f081bf50d81bf87cc23d3f3.jpg temp/1747467629_3941189_1359079433_28fe039b5717a0f08afe67c47fe4f4ae.jpg temp/1747467629_3941189_1359079431_5987c8deca6c7c21289c7d6099597d10.jpg temp/1747467629_3941189_1359079429_6a23f322dca63f8ef85c66e8db9c24d6.jpg temp/1747467629_3941189_1359079427_3176558339add313c4e6dc303d39a582.jpg temp/1747467629_3941189_1359079425_bedb4b0f7d62d0af310053fcc7afd1f3.jpg temp/1747467629_3941189_1359079423_7fac833e947adce155d111a8650e554d.jpg temp/1747467629_3941189_1359079414_d71d8a2145c307b80682826305018582.jpg temp/1747467629_3941189_1359079412_d9ca34d21b86a0aa931a346581e1cfa5.jpg temp/1747467629_3941189_1359079410_e4a8c4c10cd37c95b8c8018123281a3a.jpg temp/1747467629_3941189_1359079408_64a87aa03d59a5c2412ffa77bc0166ef.jpg temp/1747467629_3941189_1359079406_c6bd91dcdb2b67795344ca8b45b7b70e.jpg temp/1747467629_3941189_1359079404_a86ae91b215cca7cb8781b43a470dcd5.jpg temp/1747467629_3941189_1359079392_b3975a7fe05486440ecd68103455d704.jpg temp/1747467629_3941189_1359079391_c1d0fd44278d76c36f318cfd3bc17852.jpg temp/1747467629_3941189_1359079389_4dc62c1983fcd90b9151b4db8c7789c8.jpg temp/1747467629_3941189_1359079387_7c68adeef3b3de01d4d768d43a718aa3.jpg temp/1747467629_3941189_1359079386_1ccd5b4829e6c7951919e2f348be2f3b.jpg temp/1747467629_3941189_1359079384_c651334452eb26648c1b8feca5c532bc.jpg temp/1747467629_3941189_1359079375_92d60cbe1293267de4e240395e84c895.jpg temp/1747467629_3941189_1359079373_16e79c2c8fb87adb28de6947eaca67af.jpg temp/1747467629_3941189_1359079371_e7a6cd75f7ba17f5bf65f1c7ad6d12c3.jpg temp/1747467629_3941189_1359079369_50cf13f17afc5f7adbddb9dbe527dbcd.jpg temp/1747467629_3941189_1359079367_4c6af668839071d6862eaf7982810838.jpg 200it [00:00, 1217.65it/s]2025-05-17 09:41:41.498319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-17 09:41:41.697043: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-17 09:41:41.922235: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-17 09:41:41.924184: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1747467629_3941189_1359079365_c541728e72c2912f732dcb2ecf825f65.jpg temp/1747467629_3941189_1359079353_697a7f8c930af891d9397b03c2ad5319.jpg temp/1747467629_3941189_1359079351_19bf577571a1ba2787eb6818d39e1272.jpg temp/1747467629_3941189_1359079350_588814a6610ef1c8dbdafcac268226f7.jpg temp/1747467629_3941189_1359079348_9f8bde0f6a8e3fe9a6f4a89dab8b982d.jpg temp/1747467629_3941189_1359079347_d8298587134bdfce26ba41ffb5ce02c6.jpg temp/1747467629_3941189_1359079345_7c780f4703253a7348e575306f54769e.jpg temp/1747467629_3941189_1359079331_baf8bda0c026c4510465904c78a446e5.jpg temp/1747467629_3941189_1359079329_b58f0bbd1de5dc67e7e0e30d794f86ca.jpg temp/1747467629_3941189_1359079327_7bca549257bbdf638668a3a6d71a17cd.jpg temp/1747467629_3941189_1359079325_157a7e40953986df88d0af4c2841bafe.jpg temp/1747467629_3941189_1359079323_df3de07dc5bc31cf7d27badc5624e8a6.jpg temp/1747467629_3941189_1359079322_fa5acbc83ce47d007f4132d3420b5e74.jpg temp/1747467629_3941189_1359079298_6605ee33b0a04083972657a2516e637f.jpg temp/1747467629_3941189_1359079296_b82f08712228cb47d1c4ec5e55cf6c11.jpg temp/1747467629_3941189_1359079294_36668fc6263abfb910aea9ad1c561f48.jpg temp/1747467629_3941189_1359079292_838ef34d66a9b5cc274a5db5b06b113b.jpg temp/1747467629_3941189_1359079290_b3510104c71a9df2c8a669c6a5dcbe43.jpg temp/1747467629_3941189_1359079289_17c9e5175bb4a2ba0f607eb9a56b1b82.jpg temp/1747467629_3941189_1359079265_7164e714526f33c9082eefab78d9dd8a.jpg temp/1747467629_3941189_1359079264_856b5c337126f340638e25d29ad3c335.jpg temp/1747467629_3941189_1359079262_02260707978f7d3a91de6e1b74bf89a2.jpg temp/1747467629_3941189_1359079260_1b28fbee9d38c4018b972220a231c91b.jpg temp/1747467629_3941189_1359079259_0183cdfabe5a397408d99ff52f9668c8.jpg temp/1747467629_3941189_1359079257_b16404d48e09b263298ffcd0136333d7.jpg temp/1747467629_3941189_1359079245_69f07b69f5ffba58f5bbb6356e81fb0f.jpg temp/1747467629_3941189_1359079242_7362aa21f51e27f451c57e02bb777da4.jpg temp/1747467629_3941189_1359079239_958eec97f101a97650393be63c229135.jpg temp/1747467629_3941189_1359079235_750632eb914719614eb05ec9f904fb91.jpg temp/1747467629_3941189_1359079234_1ce153e76886a6caa9d81e51bad72735.jpg temp/1747467629_3941189_1359079232_62937a33fade0244ccd061fee678cab9.jpg temp/1747467629_3941189_1359079222_1aab19e6e5abd5a5538cb0e89d0f819e.jpg temp/1747467629_3941189_1359079221_3777410d122c0c41dc8f13e36cbaa2a4.jpg temp/1747467629_3941189_1359079219_dc627b35175be6fdbd0df83b91f7ec39.jpg temp/1747467629_3941189_1359079214_68bad42eb5ab236811072d3a36a5b8d9.jpg temp/1747467629_3941189_1359079209_bf9650e088ca7f08bb95420a6b5d18be.jpg temp/1747467629_3941189_1359079208_89a68edccc6bd7a3b2000e47382c5f1b.jpg temp/1747467629_3941189_1359079199_567a99e72d5d61d6ae352c27b67d60d8.jpg temp/1747467629_3941189_1359079198_135d16dc9d4bff115de66e5459f79cab.jpg temp/1747467629_3941189_1359079197_cd84814f0fa9ff7a4e61984655761ff1.jpg temp/1747467629_3941189_1359079196_35513050a3fa86efde394ad93b4a1714.jpg temp/1747467629_3941189_1359079195_3791d98b5e7c73892163912408d66f90.jpg temp/1747467629_3941189_1359079192_bdec1f00bb280ddd24deafe816901108.jpg temp/1747467629_3941189_1359079166_f0b18c1b4c2c0c41f1a570006558687f.jpg temp/1747467629_3941189_1359079163_3cffe6a8adcc5710205a3fed6933a273.jpg temp/1747467629_3941189_1359079160_2ddf085882864158af03a7ed5989f6e1.jpg temp/1747467629_3941189_1359079158_28dd66895fec23f32391ba7f137ff78a.jpg temp/1747467629_3941189_1359079156_5574b8ec5e55513e3f783330dc0f5a69.jpg temp/1747467629_3941189_1359079154_8b1b568bbbcab67c4ab11347588a4741.jpg temp/1747467629_3941189_1359079140_b7874d8e400563110332900bb93c13da.jpg temp/1747467629_3941189_1359079138_c16cce5ecb2f247a6a1d50b06c022f44.jpg temp/1747467629_3941189_1359079136_ad94bb7a1c62c186b9b0386842767111.jpg temp/1747467629_3941189_1359079135_fe24bc1090157307d7aaed2c78e67851.jpg temp/1747467629_3941189_1359079134_6556c5b325a1a8c1546ffc59d4ca91eb.jpg temp/1747467629_3941189_1359079132_550df0a5d1b8601507c435f25de2cb26.jpg temp/1747467629_3941189_1359079112_440c8a9ca07e9e027a8b4a8a927b2407.jpg temp/1747467629_3941189_1359079110_79c65dcb4d8ac49a11b6350882579ec3.jpg temp/1747467629_3941189_1359079108_1e67b39b83e319a98b2f86cce9d4ac43.jpg temp/1747467629_3941189_1359079105_960d485141f6b8ed0c876a49f8ae8d17.jpg temp/1747467629_3941189_1359079103_bc43744807e10d46a01fa0e04b2df366.jpg temp/1747467629_3941189_1359079102_778dda1b75cc5c0e82ca6132876ccbff.jpg temp/1747467629_3941189_1359079092_f9a092f66571fc1b0e18fc5668783a79.jpg temp/1747467629_3941189_1359079091_466fbd98eeb920a5244da628e2c25944.jpg temp/1747467629_3941189_1359079090_dd2768bad6d017194c93b59ac2839724.jpg temp/1747467629_3941189_1359079088_18ea08b55a8aca4365bc8f77529291a8.jpg temp/1747467629_3941189_1359079086_b7f322eedc9d1677b86cd9083c281384.jpg temp/1747467629_3941189_1359079083_d8d2d8ad64c0d8eeb5e5d09909d4cd97.jpg temp/1747467629_3941189_1359079063_192c5512148f3dbd9301e916d71e68b7.jpg temp/1747467629_3941189_1359079060_6ce64cabf73b484941137a028f1d3647.jpg temp/1747467629_3941189_1359079057_199057ca2ae3513739f9fda7ae60d52e.jpg temp/1747467629_3941189_1359079053_ccaa8ce58778760ede630d7016d07e44.jpg temp/1747467629_3941189_1359079052_f9cea06506f0acb43579af8799154a0e.jpg temp/1747467629_3941189_1359079051_b9964776f4961ff0c9be215cbfeea792.jpg temp/1747467629_3941189_1359079038_b9dc356ab20989a42594efceb806ebdb.jpg temp/1747467629_3941189_1359079036_a4cb81d20ee7559ac38590208c837002.jpg temp/1747467629_3941189_1359079034_58781730b95bee93dafcb414520cd7c5.jpg temp/1747467629_3941189_1359079032_2033b314416e67ec0b453919a1a400ae.jpg temp/1747467629_3941189_1359079031_a3643633314378ba676f168f55202e99.jpg temp/1747467629_3941189_1359079029_1995bc1f7fba02485f59fc1a39d4a864.jpg temp/1747467629_3941189_1359079019_184567e52b5ce0a20c03c9b1a26951f2.jpg temp/1747467629_3941189_1359079017_52a9ec4af6e322c0e96d0827a3159138.jpg temp/1747467629_3941189_1359079015_a29f45c208ac5221aec4275b13d7b9f9.jpg temp/1747467629_3941189_1359079013_ae81f74be5b7ad18a314ca87462b6a43.jpg temp/1747467629_3941189_1359079012_4a908c9e0beae92b3460a2bd90015d4e.jpg temp/1747467629_3941189_1359079010_98654a33cbb88fd1ecade4855ddf6d92.jpg temp/1747467629_3941189_1359078998_9ec991cf2ec23b47ca74d0009df5052f.jpg temp/1747467629_3941189_1359078996_bd3fddf9ea74cc91253b5f9e85e63200.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1359086060, 1359086058, 1359086057, 1359086056, 1359086001, 1359085997, 1359085969, 1359085965, 1359085961, 1359085956, 1359085955, 1359085950, 1359085881, 1359085877, 1359085873, 1359085871, 1359085870, 1359085868, 1359085852, 1359085851, 1359085850, 1359085849, 1359085848, 1359085846, 1359085830, 1359085829, 1359085799, 1359085798, 1359085797, 1359085796, 1359079921, 1359079917, 1359079913, 1359079909, 1359079906, 1359079902, 1359079886, 1359079882, 1359079879, 1359079876, 1359079875, 1359079873, 1359079858, 1359079855, 1359079854, 1359079850, 1359079848, 1359079845, 1359079830, 1359079828, 1359079825, 1359079822, 1359079820, 1359079817, 1359079806, 1359079803, 1359079800, 1359079796, 1359079794, 1359079791, 1359079775, 1359079772, 1359079769, 1359079766, 1359079765, 1359079763, 1359079736, 1359079734, 1359079733, 1359079731, 1359079730, 1359079728, 1359079713, 1359079712, 1359079710, 1359079708, 1359079706, 1359079702, 1359079686, 1359079684, 1359079682, 1359079679, 1359079678, 1359079677, 1359079452, 1359079450, 1359079448, 1359079446, 1359079445, 1359079443, 1359079433, 1359079431, 1359079429, 1359079427, 1359079425, 1359079423, 1359079414, 1359079412, 1359079410, 1359079408, 1359079406, 1359079404, 1359079392, 1359079391, 1359079389, 1359079387, 1359079386, 1359079384, 1359079375, 1359079373, 1359079371, 1359079369, 1359079367, 1359079365, 1359079353, 1359079351, 1359079350, 1359079348, 1359079347, 1359079345, 1359079331, 1359079329, 1359079327, 1359079325, 1359079323, 1359079322, 1359079298, 1359079296, 1359079294, 1359079292, 1359079290, 1359079289, 1359079265, 1359079264, 1359079262, 1359079260, 1359079259, 1359079257, 1359079245, 1359079242, 1359079239, 1359079235, 1359079234, 1359079232, 1359079222, 1359079221, 1359079219, 1359079214, 1359079209, 1359079208, 1359079199, 1359079198, 1359079197, 1359079196, 1359079195, 1359079192, 1359079166, 1359079163, 1359079160, 1359079158, 1359079156, 1359079154, 1359079140, 1359079138, 1359079136, 1359079135, 1359079134, 1359079132, 1359079112, 1359079110, 1359079108, 1359079105, 1359079103, 1359079102, 1359079092, 1359079091, 1359079090, 1359079088, 1359079086, 1359079083, 1359079063, 1359079060, 1359079057, 1359079053, 1359079052, 1359079051, 1359079038, 1359079036, 1359079034, 1359079032, 1359079031, 1359079029, 1359079019, 1359079017, 1359079015, 1359079013, 1359079012, 1359079010, 1359078998, 1359078996] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 1.6250107288360596 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 39.687806367874146 time spend to save output : 1.6358344554901123 total time spend for step 0 : 41.32364082336426 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.09user 5.01system 1:18.91elapsed 24%CPU (0avgtext+0avgdata 2076780maxresident)k 46952inputs+342912outputs (34major+685417minor)pagefaults 0swaps