python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2286814 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 285>100 please execute split_portfolio.py -i 24536194 -l 100 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 533, datou_cur_ids : ['3231083'] with mtr_portfolio_ids : ['24536194'] and first list_photo_ids : [] new path : /proc/2286814/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 13.687323570251465 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu Jul 3 02:16:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2203 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1984 wait 20 seconds inside check gpu memory 2025-07-03 02:17:26.057418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-07-03 02:17:26.092070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-03 02:17:26.092503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-03 02:17:26.094790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-03 02:17:26.106746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-03 02:17:26.107467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-03 02:17:26.123461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-03 02:17:26.125790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-03 02:17:26.160850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-03 02:17:26.162965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-03 02:17:26.163584: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-07-03 02:17:26.199407: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493035000 Hz 2025-07-03 02:17:26.202056: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f765c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-07-03 02:17:26.202125: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-07-03 02:17:26.335372: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3528e640 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-07-03 02:17:26.335454: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-07-03 02:17:26.336883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-03 02:17:26.337039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-03 02:17:26.337073: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-03 02:17:26.337109: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-03 02:17:26.337137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-03 02:17:26.337165: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-03 02:17:26.337191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-03 02:17:26.337226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-03 02:17:26.338730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-03 02:17:26.338836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-03 02:17:26.339820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-07-03 02:17:26.339859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-07-03 02:17:26.339879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-07-03 02:17:26.341417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3175 max_wait_temp : 3 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-07-03 02:17:34.365387: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.976315021514893 time used to load_weights : 0.7785682678222656 0it [00:00, ?it/s] 51it [00:00, 496.08it/s] 100it [00:00, 513.04it/s]2025-07-03 02:17:39.498122: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-03 02:17:39.798390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-03 02:17:39.980366: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-07-03 02:17:40.014638: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1751501786_2286814_1370227529_39a6b45dc351bea876aa08323e9992fc_6165027.jpg temp/1751501786_2286814_1370227526_9f7c851fff756c77fff3f6e166814868_73015162.jpg temp/1751501786_2286814_1370227523_c81b34e1adb9f2ce17a81bac93fef438_24085246.jpg temp/1751501786_2286814_1370227521_9b3027d0c7c67e6292b8e5fba58caedd_57041490.jpg temp/1751501786_2286814_1370227519_d4782c9b3ca3b8a01db79789c494dc86_6557426.jpg temp/1751501786_2286814_1370227517_2f7b3613eefdd6e1c57ad8434ea48632_7355851.jpg temp/1751501786_2286814_1370227189_9c248772f03d9b99d7cfdb12f1f26a1a_56756207.jpg temp/1751501786_2286814_1370227187_fb6103cef7ab6768e7083fd4c93d03a7_44984344.jpg temp/1751501786_2286814_1370227185_51092f26a073b3724737f37c78f23569_79079691.jpg temp/1751501786_2286814_1370227183_c8052f832b2e677d7a03e25c01831015_81610630.jpg temp/1751501786_2286814_1370227181_4e85568d11c0c148b74f9a6d09199e94_77189680.jpg temp/1751501786_2286814_1370227179_f29d377c490e3537d5805d3c34e70fe5_43377732.jpg temp/1751501786_2286814_1370226947_6bdd18e1a9579e7f64cb6c90b6b61b76_4877401.jpg temp/1751501786_2286814_1370226943_bf10da07f6955bba7e096294f93faad2_82640716.jpg temp/1751501786_2286814_1370226939_3ee87202fb9af750cebfeba43344c142_81589036.jpg temp/1751501786_2286814_1370226934_3caa7fd016fb576d5d75f3355cb59830_64175384.jpg temp/1751501786_2286814_1370226930_542b9c82cdd7178131b3da32ea397c0f_70504064.jpg temp/1751501786_2286814_1370226926_78bd6558d24260d9219869d321467ff7_50035145.jpg temp/1751501786_2286814_1370226657_78bfbd2a26ac7652be19f1e05c30180c_32506777.jpg temp/1751501786_2286814_1370226655_fbddf346a4efe42f694285e47937be17_14252190.jpg temp/1751501786_2286814_1370226653_7742ad14a36ae96a0f2a3c7d3e75f824_89049375.jpg temp/1751501786_2286814_1370226651_c049291fcfc23b0ab0441b2e334bea69_61679850.jpg temp/1751501786_2286814_1370226650_cbec6e6a3a98cc0377c6b783d45944ec_86303270.jpg temp/1751501786_2286814_1370226649_0c0ca8ca1612e874a9e3d850e216f289_1961076.jpg temp/1751501786_2286814_1370226610_78bb2fdf834ecc3427ffeca9d613d71e_41226392.jpg temp/1751501786_2286814_1370226609_392c0eeb8255e978244bdc2839e93d74_58644598.jpg temp/1751501786_2286814_1370226608_dc0bc64eb6ed5a7efa43e7968c5cfa96_86607695.jpg temp/1751501786_2286814_1370226607_4386a62e0f9a28666b347dbc29aa2d52_66751969.jpg temp/1751501786_2286814_1370226606_f300c0d44b58f115348ce21c0372ccc3_31488956.jpg temp/1751501786_2286814_1370226605_977bc13a2c2acc9150b3f8d9fce50620_97489767.jpg temp/1751501786_2286814_1370225535_dcb7d7bc900fc79e2f5e67c767d1981a_40652136.jpg temp/1751501786_2286814_1370225533_7f4655cce9406b42c01d5138b9da4442_18666425.jpg temp/1751501786_2286814_1370225531_ddf97291d2c4ef07ea7d936f365addb4_17710087.jpg temp/1751501786_2286814_1370225529_b02f232fa70a231a1c1564537c78a38b_40627579.jpg temp/1751501786_2286814_1370225527_9e062d6443700636ceac59228dcc430f_51805511.jpg temp/1751501786_2286814_1370225525_3279db3c52e4f2b4c0f3b522f82b05c4_82028623.jpg temp/1751501786_2286814_1370225325_81683972c9960f36ae8dd71a3740d653_96104076.jpg temp/1751501786_2286814_1370225323_41b757ced3b21917d2cf48288d79d6ef_98236480.jpg temp/1751501786_2286814_1370225321_b89d4d95a2231beba8132e69cf4aa3a4_64074789.jpg temp/1751501786_2286814_1370225319_6d1ad3d8b32299e65886cc0740ef67c1_79633056.jpg temp/1751501786_2286814_1370225317_8a9eb915b6a7674608029460d4350e8e_50453499.jpg temp/1751501786_2286814_1370225315_2717363855527dcf12bf32933828791b_14606433.jpg temp/1751501786_2286814_1370225091_4dee412189d4346e66580f88cad8eb03_88306262.jpg temp/1751501786_2286814_1370225090_3123399086fd3e7b02730ebec69cf80c_77161865.jpg temp/1751501786_2286814_1370225088_abf1ce7a18a2a510dfae08de365b1192_70460892.jpg temp/1751501786_2286814_1370225086_9ae17929c34ad078f42bdf5abbf97f39_23753842.jpg temp/1751501786_2286814_1370225084_57bf7c55d7187fe84b05dcff33437a8d_76430335.jpg temp/1751501786_2286814_1370225082_41c9bead58486053eda01bd32a15b2bc_7060126.jpg temp/1751501786_2286814_1370224785_31d9de130653a03dbfaac15bfeeb114d_28118054.jpg temp/1751501786_2286814_1370224782_c00f6bf78e27cbeef5d69f6f041664a2_33507570.jpg temp/1751501786_2286814_1370224779_ccbfde80707c01c3c86a3aa824b4ed82_34880389.jpg temp/1751501786_2286814_1370224776_6bd2b40d54f338ef83e701c50820b3ab_63694408.jpg temp/1751501786_2286814_1370224773_e1c8648c00ea0c4d26b51d1243a91b50_9409197.jpg temp/1751501786_2286814_1370224770_ddf29dd258e0dc2e53ea00ce8eeef463_71710011.jpg temp/1751501786_2286814_1370224205_554d1f3fedff0c4ebc2d51a04203b6aa_35686075.jpg temp/1751501786_2286814_1370224201_07bfdb4119b88d723b2f9dee5f44a95a_40325898.jpg temp/1751501786_2286814_1370224197_6c1110e2b4f37eedd1f68f59016d8b32_96482083.jpg temp/1751501786_2286814_1370224193_27ef04f78de63c0a6d67400e79febea5_63598367.jpg temp/1751501786_2286814_1370224190_f1e63903bf380da07ef230062145bf42_11455449.jpg temp/1751501786_2286814_1370224187_7f5bc1d58b1cc856c79af94373ee7b2d_89750000.jpg temp/1751501786_2286814_1370222947_fccf926b9dbc60c00544361c4b1b99b8_343574.jpg temp/1751501786_2286814_1370222943_185adf6e6e30d4ef1ea66484ebf131a4_7030218.jpg temp/1751501786_2286814_1370222939_a9dbc8768f3bda902dc3714e40d442cf_33347328.jpg temp/1751501786_2286814_1370222936_9426ebea4eee2fb1b3faacbd872d0e43_3716423.jpg temp/1751501786_2286814_1370222933_1e5c5aa106039bc0467bd28b31c3a9f6_33877474.jpg temp/1751501786_2286814_1370222931_ce5c60156c6e20cdcc5935aec539a9ec_27094949.jpg temp/1751501786_2286814_1370221090_cab27e6e0c548e046f1e89f42e2d0f55_24895548.jpg temp/1751501786_2286814_1370221088_805e50f47e9c0b01bf66ac4f7a5cb2cd_74947496.jpg temp/1751501786_2286814_1370221086_91ffb23272d680779cf14b708d099661_1856084.jpg temp/1751501786_2286814_1370221084_767995a86c382ab96aa1c6708d0a98f2_27538883.jpg temp/1751501786_2286814_1370221082_81ab811696ec895ed4adc6cc2e5a0161_79756598.jpg temp/1751501786_2286814_1370221080_80301eb4a23e94d2b4a9d57d77caa9fb_50188215.jpg temp/1751501786_2286814_1370219402_c6ddd8692ec23476ac12842715f745c5_95293602.jpg temp/1751501786_2286814_1370219401_346b5383fb17ff53ce35bab77c49f230_95067049.jpg temp/1751501786_2286814_1370219399_e1ddbf6b825707b9e94e0a9f5781634f_62190892.jpg temp/1751501786_2286814_1370219398_bbaccd938dafce4311d4666c860d8920_2698501.jpg temp/1751501786_2286814_1370219397_a63cf4d591a22bfa5ddb1cbac581cf6b_66965870.jpg temp/1751501786_2286814_1370219396_ed1733c06931317cbc19219603f2fb08_28209800.jpg temp/1751501786_2286814_1370218897_5541d484998739c99514a61e6737f96e_60586033.jpg temp/1751501786_2286814_1370218895_c2828353b8e3a4113661b134223725ff_74961412.jpg temp/1751501786_2286814_1370218893_375067b336f98dce2eb7b8d32da58386_35724822.jpg temp/1751501786_2286814_1370218891_2300df4ba530ae91869fe1f7f4709c05_34502437.jpg temp/1751501786_2286814_1370218889_3302cc705cd06bd959911ad8954f8d8c_89572517.jpg temp/1751501786_2286814_1370218887_74711d919bfa6f90e02a54b8d994d035_15811709.jpg temp/1751501786_2286814_1370218460_f6b8730ec3901ae8cbbc4c7c080f99b1_13186230.jpg temp/1751501786_2286814_1370218456_1b7ebf841e44260754e497b0e0c014c5_92592000.jpg temp/1751501786_2286814_1370218453_a63498cc448089bd79db8dcb85b95679_43510486.jpg temp/1751501786_2286814_1370218450_7f37bae5c6b38812dca3a9c6b6692009_15023549.jpg temp/1751501786_2286814_1370218447_2e2d655afc1791e3a1191a46ead10337_82611076.jpg temp/1751501786_2286814_1370218444_6bf0970cd6739571f006ba5006a3e4c1_175228.jpg temp/1751501786_2286814_1370216655_17afad0dd43df81d599f82260cd9c763_77706081.jpg temp/1751501786_2286814_1370216654_9e71b094249d67e8b4fb7df53663b5bb_97391945.jpg temp/1751501786_2286814_1370216653_2ce214df23a0d9b16ea34769a2d63622_42598400.jpg temp/1751501786_2286814_1370216651_65c0567c48fdb7ffa5d6a05618f4d20d_33314174.jpg temp/1751501786_2286814_1370216649_755ab8be92e2a24f49b1510d6ff37f90_80508956.jpg temp/1751501786_2286814_1370216646_661fc0d537e6e4b40e5fe03436cf5ae5_95325263.jpg temp/1751501786_2286814_1370215703_b571ce224b4aef1686502ba288f151d1_21377289.jpg temp/1751501786_2286814_1370215702_48bc56bc80909cb6774868650ed3a255_71619280.jpg temp/1751501786_2286814_1370215701_3a2048c0b66af0f834a6457bb53d6ed9_73282117.jpg temp/1751501786_2286814_1370215700_43b9684b0aeaa84f55bc8eb032d46a08_42479921.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1370227529, 1370227526, 1370227523, 1370227521, 1370227519, 1370227517, 1370227189, 1370227187, 1370227185, 1370227183, 1370227181, 1370227179, 1370226947, 1370226943, 1370226939, 1370226934, 1370226930, 1370226926, 1370226657, 1370226655, 1370226653, 1370226651, 1370226650, 1370226649, 1370226610, 1370226609, 1370226608, 1370226607, 1370226606, 1370226605, 1370225535, 1370225533, 1370225531, 1370225529, 1370225527, 1370225525, 1370225325, 1370225323, 1370225321, 1370225319, 1370225317, 1370225315, 1370225091, 1370225090, 1370225088, 1370225086, 1370225084, 1370225082, 1370224785, 1370224782, 1370224779, 1370224776, 1370224773, 1370224770, 1370224205, 1370224201, 1370224197, 1370224193, 1370224190, 1370224187, 1370222947, 1370222943, 1370222939, 1370222936, 1370222933, 1370222931, 1370221090, 1370221088, 1370221086, 1370221084, 1370221082, 1370221080, 1370219402, 1370219401, 1370219399, 1370219398, 1370219397, 1370219396, 1370218897, 1370218895, 1370218893, 1370218891, 1370218889, 1370218887, 1370218460, 1370218456, 1370218453, 1370218450, 1370218447, 1370218444, 1370216655, 1370216654, 1370216653, 1370216651, 1370216649, 1370216646, 1370215703, 1370215702, 1370215701, 1370215700] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.06625795364379883 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 59.81499624252319 time spend to save output : 0.07645511627197266 total time spend for step 0 : 59.891451358795166 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 15.54user 5.63system 1:17.45elapsed 27%CPU (0avgtext+0avgdata 2033556maxresident)k 1003904inputs+283096outputs (3464major+636947minor)pagefaults 0swaps