python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3754618 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3962737'] with mtr_portfolio_ids : ['27928813'] and first list_photo_ids : [] new path : /proc/3754618/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 97 ; length of list_pids : 97 ; length of list_args : 97 time to download the photos : 19.481826305389404 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu Oct 30 01:00:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-10-30 01:00:51.990486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-10-30 01:00:51.999429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-30 01:00:51.999700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-30 01:00:52.001801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-30 01:00:52.004093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-30 01:00:52.004441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-30 01:00:52.006742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-30 01:00:52.007839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-30 01:00:52.012561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-30 01:00:52.013622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-30 01:00:52.014130: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-10-30 01:00:52.038670: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-10-30 01:00:52.039965: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe4b4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-10-30 01:00:52.039986: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-10-30 01:00:52.171390: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x18407480 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-10-30 01:00:52.171435: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-10-30 01:00:52.172242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-30 01:00:52.172337: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-30 01:00:52.172372: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-30 01:00:52.172391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-30 01:00:52.172408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-30 01:00:52.172426: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-30 01:00:52.172443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-30 01:00:52.172461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-30 01:00:52.173250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-30 01:00:52.173296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-30 01:00:52.173843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-10-30 01:00:52.173859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-10-30 01:00:52.173867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-10-30 01:00:52.174733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3227 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-10-30 01:00:59.633084: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-10-30 01:00:59.633799: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.772907972335815 time used to load_weights : 0.2883749008178711 0it [00:00, ?it/s] 97it [00:00, 1140.46it/s]2025-10-30 01:01:03.654167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-30 01:01:03.688413: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-10-30 01:01:03.780621: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-30 01:01:03.783977: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-10-30 01:01:03.785249: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1761782429_3754618_1390209901_58cbfc5e505d6a31f5b311ee8e241fcc.jpg temp/1761782429_3754618_1390209897_9a9b298457d3db7fe568d1b1679bccda.jpg temp/1761782429_3754618_1390209771_5e536a4829fb2519a36da09f4902e9f9.jpg temp/1761782429_3754618_1390209767_6f1b82458c7f92655e478f0138af9390.jpg temp/1761782429_3754618_1390209444_d3a7b66ff99af8b36fc1ef61e0ff5b47.jpg temp/1761782429_3754618_1390209416_f2430117943d4ff1d7eb003f6e1b5ac8.jpg temp/1761782429_3754618_1390209401_3a3d5585749f858e44a3ef8cfbc5cdde.jpg temp/1761782429_3754618_1390209396_ce88b296c3b753cee8d7fa314bad0452.jpg temp/1761782429_3754618_1390209359_109455b690a191ff16a3d6071c463ea4.jpg temp/1761782429_3754618_1390209347_0b39c56a15d4a697e9ffef796e269801.jpg temp/1761782429_3754618_1390209216_a0dac1b3d7e3234cfb7bdc9dbb16039e.jpg temp/1761782429_3754618_1390209183_de3a3b57fc01cdfe516e3e5e09bbcf1f.jpg temp/1761782429_3754618_1390209056_ccd0078d5d5fae77043507a6fb38868c.jpg temp/1761782429_3754618_1390209054_fb1246f028ffbcc14cc514dd9617fbc1.jpg temp/1761782429_3754618_1390208915_f5b3976d5a378ae21bf2969225371230.jpg temp/1761782429_3754618_1390208460_78ce10a5d72e19118a1e2b5a61bf0b1c.jpg temp/1761782429_3754618_1390208275_f521126d5765d1c7958e2b23bede0bd1.jpg temp/1761782429_3754618_1390208272_18872b89a313e385c4828580844719e1.jpg temp/1761782429_3754618_1390208269_1d2633995ac21972eacf650f73dc7e34.jpg temp/1761782429_3754618_1390208266_b9f8da5a94df0627e41b8ec2817f5b76.jpg temp/1761782429_3754618_1390208260_5c7dab0a59829b22a686b16d711ae82c.jpg temp/1761782429_3754618_1390207676_01bb7cbcd67c1c9df976c501548f17af.jpg temp/1761782429_3754618_1390207635_3e4c52e9ba3810717834ac409f5ccde5.jpg temp/1761782429_3754618_1390207257_5b08130b47b6e203906068851fc47098.jpg temp/1761782429_3754618_1390207207_4a0e9626997cb20cc4b8fc8f2a6ef5c0.jpg temp/1761782429_3754618_1390206929_90aac20f1fa031574cd098a29bf038d8.jpg temp/1761782429_3754618_1390206874_643ac359464ffb0e062f5116faa2acf3.jpg temp/1761782429_3754618_1390206314_ed8bd7e9dce96b0ebadc9b987754e08b.jpg temp/1761782429_3754618_1390206246_5917763d5c9cce2587d4e651202851d1.jpg temp/1761782429_3754618_1390205704_11dd7dd9927ae075120db335cf03dd82.jpg temp/1761782429_3754618_1390205657_e4a0c0b5d037c9a7e4029af4c9c84153.jpg temp/1761782429_3754618_1390205501_f3738b61e9c88cc54d3c91401decab02.jpg temp/1761782429_3754618_1390205497_6db879c859da27b4ba6bdc24a371809a.jpg temp/1761782429_3754618_1390205449_6b5b05209c5aa41acb157dd6f29a48f2.jpg temp/1761782429_3754618_1390205446_dbfa9a46acd829bb1df999c18931ca3c.jpg temp/1761782429_3754618_1390205397_f562ec29ed5c29a0f13e683219023b05.jpg temp/1761782429_3754618_1390205392_1824f7330f3115be2368c70360f98200.jpg temp/1761782429_3754618_1390205387_9c18cf2135ffdac3c63056d30a784b53.jpg temp/1761782429_3754618_1390205383_27abec1f4d3f485fd3cfd87da80b9864.jpg temp/1761782429_3754618_1390205337_eed0f156136864a64a234d96ad0face6.jpg temp/1761782429_3754618_1390205333_bc8df3d6f2384d24ea80f7050a5e71db.jpg temp/1761782429_3754618_1390205329_10a511c0df7d1f81834ca4fbadab2ef3.jpg temp/1761782429_3754618_1390205159_727f6ba4e28aa529c141ec4c1fb5c53f.jpg temp/1761782429_3754618_1390205157_25e4a032c9d382fb21ff01d76cc8010f.jpg temp/1761782429_3754618_1390205111_33a445fb78b7fce348440b937d97d425.jpg temp/1761782429_3754618_1390205105_55e5a378c68006df8261b7be8cfae4e1.jpg temp/1761782429_3754618_1390205099_47437f84c9d13af278550db4c2dbe433.jpg temp/1761782429_3754618_1390205095_a525840a617de24c2af6536e36cdf456.jpg temp/1761782429_3754618_1390205025_4a41f2af2adf7b17b1a49b37f5cc23ec.jpg temp/1761782429_3754618_1390205023_230566b73ccb60e63f2c76d58575cac0.jpg temp/1761782429_3754618_1390204983_7ab35530eb55613f96751c8903624347.jpg temp/1761782429_3754618_1390204978_4c21e084403d304e027da7ba3597cc0a.jpg temp/1761782429_3754618_1390204974_66a8d2398efed9ec53e4eb68a37877d8.jpg temp/1761782429_3754618_1390204971_9a9d9db6c83854d5af34daa5e949be4d.jpg temp/1761782429_3754618_1390204969_13ce8adb1fee0189450f043fe69938dd.jpg temp/1761782429_3754618_1390204727_5f9f0db2cdd71c9cfdfb0c1e8f2b6da9.jpg temp/1761782429_3754618_1390204725_ca1822ec1acde616fa1cb06708b29809.jpg temp/1761782429_3754618_1390204562_877f056dbd9b3e6abc7710b263b8cfa3.jpg temp/1761782429_3754618_1390204543_f00a5a55e14f2d0ca2d5a9d2bdeca6a0.jpg temp/1761782429_3754618_1390204325_b947951eb025aecd0c58d427cb3f510b.jpg temp/1761782429_3754618_1390204320_3e5b885e5c71a3c570dd46ed8f794265.jpg temp/1761782429_3754618_1390204311_9c47c7a5d0598a56faf707cd4616e47e.jpg temp/1761782429_3754618_1390204308_079e105a8167a19987d11be022ca1d64.jpg temp/1761782429_3754618_1390204129_717b0d27c3a145538625d359044472ac.jpg temp/1761782429_3754618_1390204125_ab41e0e6a538b6628e6b167bc1ceadf9.jpg temp/1761782429_3754618_1390203895_cd8de7a0672591ef0333accc0e30b5ed.jpg temp/1761782429_3754618_1390203872_1759cd07e7529c92459338b7cbd32d53.jpg temp/1761782429_3754618_1390203852_fbc0f2c8944bf5f62b7162115a12592a.jpg temp/1761782429_3754618_1390203849_2126648f7fe03207007f86a431176096.jpg temp/1761782429_3754618_1390203694_53a26da61b5b88eb46be5f85b049a764.jpg temp/1761782429_3754618_1390203683_94387beb00c888cc20d2b2d0a80a343c.jpg temp/1761782429_3754618_1390203652_d909e884c813313f472d37472c0e058b.jpg temp/1761782429_3754618_1390203648_e925aaa9b80c69b13b8b4467617fe7f5.jpg temp/1761782429_3754618_1390203391_f153374572b85435e3de8eb831063abc.jpg temp/1761782429_3754618_1390203368_5802339068f14f80d7c5a25101a0862d.jpg temp/1761782429_3754618_1390203318_ee0a44b9e59062b31bab03447995e148.jpg temp/1761782429_3754618_1390203313_e5762647737add589e7f9ed7f445acf2.jpg temp/1761782429_3754618_1390203303_cf384d7a37d92fc31f6a2edc35695913.jpg temp/1761782429_3754618_1390203295_bef0325ee9b52b7d46e8ccfc241b1590.jpg temp/1761782429_3754618_1390203027_ce9cee490897d0b3b3b8dafab495e7cd.jpg temp/1761782429_3754618_1390202969_f8b0d1d28658c81679b29d31c8775219.jpg temp/1761782429_3754618_1390202837_868ef047cf489da2593be165b94282e9.jpg temp/1761782429_3754618_1390202807_0f6a4f3f27fc6aa9fa403c3570eac67d.jpg temp/1761782429_3754618_1390202557_975cb5ecde9671d7d0d9c8a9e70137c3.jpg temp/1761782429_3754618_1390202554_23a7a9b783af12b2d0e28633850ab07a.jpg temp/1761782429_3754618_1390202530_af96bee2fa746dfb0c1f4a15ba8f1072.jpg temp/1761782429_3754618_1390202526_9b124e4e98ebbd6aebe52fb65c8b5e34.jpg temp/1761782429_3754618_1390202420_780fb8769d2023c507f56e62aa828b4f.jpg temp/1761782429_3754618_1390202416_ebd19f279b8bf7ee211792de856fcc91.jpg temp/1761782429_3754618_1390202303_b6e58dc6c9c134743fb82f83a17c5eb0.jpg temp/1761782429_3754618_1390202299_2ebf4a598f1997a206e2687b2d2cb245.jpg temp/1761782429_3754618_1390202229_8aaaa4bb55b4a1e2c0b19d2b334cec0c.jpg temp/1761782429_3754618_1390202226_d1a88a314d235ed52e1a55511c6df1b4.jpg temp/1761782429_3754618_1390202178_c5e1c327799462818e3117f955dff4f7.jpg temp/1761782429_3754618_1390202177_8fdcc142de1f00d6261b7e5e681003ee.jpg temp/1761782429_3754618_1390202129_7d14517d07a6e3a25d4c417f1c856f1d.jpg temp/1761782429_3754618_1390202124_3aecebe20d233d4a080a37e2710af02f.jpg Found 97 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2339, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2533, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1390209901, 1390209897, 1390209771, 1390209767, 1390209444, 1390209416, 1390209401, 1390209396, 1390209359, 1390209347, 1390209216, 1390209183, 1390209056, 1390209054, 1390208915, 1390208460, 1390208275, 1390208272, 1390208269, 1390208266, 1390208260, 1390207676, 1390207635, 1390207257, 1390207207, 1390206929, 1390206874, 1390206314, 1390206246, 1390205704, 1390205657, 1390205501, 1390205497, 1390205449, 1390205446, 1390205397, 1390205392, 1390205387, 1390205383, 1390205337, 1390205333, 1390205329, 1390205159, 1390205157, 1390205111, 1390205105, 1390205099, 1390205095, 1390205025, 1390205023, 1390204983, 1390204978, 1390204974, 1390204971, 1390204969, 1390204727, 1390204725, 1390204562, 1390204543, 1390204325, 1390204320, 1390204311, 1390204308, 1390204129, 1390204125, 1390203895, 1390203872, 1390203852, 1390203849, 1390203694, 1390203683, 1390203652, 1390203648, 1390203391, 1390203368, 1390203318, 1390203313, 1390203303, 1390203295, 1390203027, 1390202969, 1390202837, 1390202807, 1390202557, 1390202554, 1390202530, 1390202526, 1390202420, 1390202416, 1390202303, 1390202299, 1390202229, 1390202226, 1390202178, 1390202177, 1390202129, 1390202124] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 97 time used for this insertion : 0.043894052505493164 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 14.589352369308472 time spend to save output : 0.05419182777404785 total time spend for step 0 : 14.64354419708252 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.45user 4.47system 0:39.99elapsed 47%CPU (0avgtext+0avgdata 1832920maxresident)k 449624inputs+238080outputs (272major+619854minor)pagefaults 0swaps