python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2924570 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7614>100 please execute split_portfolio.py -i 23315442 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2947888'] with mtr_portfolio_ids : ['23315442'] and first list_photo_ids : [] new path : /proc/2924570/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 17.40746307373047 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Wed May 28 00:49:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-28 00:49:48.911090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-28 00:49:48.943738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-28 00:49:48.944049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 00:49:48.946268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 00:49:48.948381: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-28 00:49:48.948731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-28 00:49:48.950785: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-28 00:49:48.951867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-28 00:49:48.955996: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 00:49:48.957062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-28 00:49:48.957377: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-28 00:49:48.983115: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-28 00:49:48.985168: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff9c8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-28 00:49:48.985213: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-28 00:49:49.208793: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23d33860 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-28 00:49:49.208840: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-28 00:49:49.209912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-28 00:49:49.210010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 00:49:49.210044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 00:49:49.210074: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-28 00:49:49.210104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-28 00:49:49.210134: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-28 00:49:49.210164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-28 00:49:49.210193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 00:49:49.211657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-28 00:49:49.211737: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 00:49:49.212621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-28 00:49:49.212645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-28 00:49:49.212660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-28 00:49:49.214236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3697 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.2665698528289795 time used to load_weights : 0.5524411201477051 0it [00:00, ?it/s] 100it [00:00, 1764.40it/s]2025-05-28 00:49:58.753247: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 00:49:58.947453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 00:49:59.573838: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-28 00:49:59.601027: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748386168_2924570_1360988430_ff06b38187d8cfb2c97915e12c68ecc0.jpg temp/1748386168_2924570_1360988427_4ad1e6cc59387ed542a9fa707e30304e.jpg temp/1748386168_2924570_1360988425_d473cc9fb1f5d0aa9be268a9be6cdcbd.jpg temp/1748386168_2924570_1360988423_4a7f04010dd3e2a657135af4c6693e63.jpg temp/1748386168_2924570_1360988420_e0b53c30cdda3a78f3a9b8bfa7ff7d7c.jpg temp/1748386168_2924570_1360988418_88e0c3357422f7c37e081b9981766018.jpg temp/1748386168_2924570_1360988397_4bf971e7f58a7da52280b0d3a8998c92.jpg temp/1748386168_2924570_1360988395_e98f84bf025bc934a411e7a3e6d25a69.jpg temp/1748386168_2924570_1360988394_59e6fe7f2221baf3dfcdca49c5d47d74.jpg temp/1748386168_2924570_1360988393_7a1da1690fc21d209eb6e89cbe215442.jpg temp/1748386168_2924570_1360988391_e8b00d16c2a30040ea4313b7a19639a8.jpg temp/1748386168_2924570_1360988389_c34322cc9c762e4ad6372272d20c31d6.jpg temp/1748386168_2924570_1360988370_b2445a714631ad669d94539c87dde2ce.jpg temp/1748386168_2924570_1360988367_2733b73b859a51dc77240ada6812720c.jpg temp/1748386168_2924570_1360988365_64f1092dba898ce20c610cb746306da2.jpg temp/1748386168_2924570_1360988364_1ade59be8ea32cccec9a442fc5a41bc9.jpg temp/1748386168_2924570_1360988363_467b79d7a8b7aee1f86f6210f64bc56d.jpg temp/1748386168_2924570_1360988362_92b627d4f94e49b885c92e3fb1269956.jpg temp/1748386168_2924570_1360988331_5f6a56c20bd284fcda9b70b526513722.jpg temp/1748386168_2924570_1360988329_ac23f960bbdac6dc5eab9da47c7cd886.jpg temp/1748386168_2924570_1360988328_4fb3364b40b4f53588dc9d26d85a3004.jpg temp/1748386168_2924570_1360988327_e9a082675af3223e2ecc456cebf91cf2.jpg temp/1748386168_2924570_1360988325_44f324cad026ef701e5a8d8278499108.jpg temp/1748386168_2924570_1360988323_4fa31f90f4bf80ebb2c0abcf91f3be08.jpg temp/1748386168_2924570_1360988310_cf628e6d069c71e4db1c5222114ce09b.jpg temp/1748386168_2924570_1360988307_bbcd97834e1e869c3bf182568d63f9c7.jpg temp/1748386168_2924570_1360988305_f1b33792d5ba2fc80ee96e5f131536f2.jpg temp/1748386168_2924570_1360988304_70d27563100d06d32bd49bcfedf6a20f.jpg temp/1748386168_2924570_1360988303_92617e0422985cdfd2dc6f2e67d9dc64.jpg temp/1748386168_2924570_1360988301_d3b2aa67640676e11250e418dd053676.jpg temp/1748386168_2924570_1360988284_45714f2d0daac159a6c289adb2b419b6.jpg temp/1748386168_2924570_1360988280_a56805baf263eaf6a10ea88c072c5a0f.jpg temp/1748386168_2924570_1360988277_6e5e4b067af210f5b3493a0ab247cf99.jpg temp/1748386168_2924570_1360988274_777df07f3e6095c3be31939576a7d5ee.jpg temp/1748386168_2924570_1360988270_1572e5ad41034f09f06eb9568884209e.jpg temp/1748386168_2924570_1360988267_6dc54c49f00de2f4c5ae54ae4498d46c.jpg temp/1748386168_2924570_1360988248_02ea3e6e00fff60ad7bf5140c44fbd3b.jpg temp/1748386168_2924570_1360988246_2d49ec281dbd58c79bbdc0ccce40af87.jpg temp/1748386168_2924570_1360988243_ad8b9d38608db384f9d72dfa71299494.jpg temp/1748386168_2924570_1360988241_5dce23e3928195c204a5f86c17d20153.jpg temp/1748386168_2924570_1360988239_56e41d56ff4f32a7cbc9965478b136ca.jpg temp/1748386168_2924570_1360988237_07081d596d0478055acf46531a2b9202.jpg temp/1748386168_2924570_1360988235_f0fa89dcb82ead2ec2fa31cac17493bb.jpg temp/1748386168_2924570_1360988233_5e353201168c8e7b769f09f7feea20c2.jpg temp/1748386168_2924570_1360988231_a15cd5a33fd7dfc6776d40b966fe5322.jpg temp/1748386168_2924570_1360988230_593ed9626cdf991253a0a1fc2cef5305.jpg temp/1748386168_2924570_1360988229_4b17edac1d49bfacc160f9e647e54ff9.jpg temp/1748386168_2924570_1360988227_8a5a1d5baa59f7aee604bc892b949581.jpg temp/1748386168_2924570_1360988212_578f6405dca8250d8665c784b6770b12.jpg temp/1748386168_2924570_1360988210_11437619826e846ca1d5bb8506cc9253.jpg temp/1748386168_2924570_1360988209_dc70bb1506ffa262294cfb263d95211e.jpg temp/1748386168_2924570_1360988208_a3986cd4fac860b1a58b230b4bdac2d8.jpg temp/1748386168_2924570_1360988206_a27eb5477f50a96c211bd55cdbafac36.jpg temp/1748386168_2924570_1360988203_887eae510a45c0d1c890e93a6e89c305.jpg temp/1748386168_2924570_1360988145_6c372d855d9a609db41925a9a789188f.jpg temp/1748386168_2924570_1360988142_6081f489c4b72fcaabe386be90788833.jpg temp/1748386168_2924570_1360988139_6c45f6d7f610d5577a0ec0d830ce307c.jpg temp/1748386168_2924570_1360988137_0a930ae9dee9adaec809e39d8a2ffff7.jpg temp/1748386168_2924570_1360988136_90c45018b08b1fd14e4980be00964928.jpg temp/1748386168_2924570_1360988133_bb943919b771915cefa7e58c76b01211.jpg temp/1748386168_2924570_1360988129_3f882164a4992d21a00e6f170e9fe150.jpg temp/1748386168_2924570_1360988127_2cddab874bc7f76b7e98bbf8ef1e6556.jpg temp/1748386168_2924570_1360988126_b6e7b37e7e7c68a6d54a15e1486aca14.jpg temp/1748386168_2924570_1360988125_9a3b95e7210b8b637fe5f010036ef8b8.jpg temp/1748386168_2924570_1360988123_891171e6df4736dd5aac97708f17a6b6.jpg temp/1748386168_2924570_1360988121_3cd80f0a25521e3d555e1fa912f6a88a.jpg temp/1748386168_2924570_1360988102_63fcdcf8ea0819a4bb4669a791ada5fe.jpg temp/1748386168_2924570_1360988100_26fb8cf843dbfdadc7f478579c15ace5.jpg temp/1748386168_2924570_1360988099_0546dda595a715edbabf01afa0b9b796.jpg temp/1748386168_2924570_1360988098_98f22a54f194f340de2c5db13a22cc60.jpg temp/1748386168_2924570_1360988096_fe8c01b0edd955fdfeaa0c2ac73588a7.jpg temp/1748386168_2924570_1360988094_93c8c902bd57d7fd5c9e63ae9ef10cf9.jpg temp/1748386168_2924570_1360988080_a54fe44fd8c5c591dba2e7222640a7ec.jpg temp/1748386168_2924570_1360988078_4956545c3e36bb43d765d8177836ceb1.jpg temp/1748386168_2924570_1360988077_113a67669042c1c07daf3253f66c8fa1.jpg temp/1748386168_2924570_1360988076_a0089c1a04b75face94c38da6c3804b4.jpg temp/1748386168_2924570_1360988075_9f8a0fa94060d8f5621c9a9b9a30b746.jpg temp/1748386168_2924570_1360988074_add392bd328f70e7289e371d6ca8a461.jpg temp/1748386168_2924570_1360988060_2c94c02f44fef7cd9c03567241e40104.jpg temp/1748386168_2924570_1360988057_b3e1a17fee2525f8e2f780b87132367d.jpg temp/1748386168_2924570_1360988055_712acca5ea92ce726a72dd0f970ef71d.jpg temp/1748386168_2924570_1360988053_8ab1d88a98f4d6d4ef55b25911d9185e.jpg temp/1748386168_2924570_1360988051_b0ee7190e7361d4d9d43f03a79aed896.jpg temp/1748386168_2924570_1360988049_8cdd936990a2b98dbcdbd46fd25b0a9b.jpg temp/1748386168_2924570_1360988022_ff043451671b828f2e15c9aeae6f1418.jpg temp/1748386168_2924570_1360988019_d6505f81d84fbc0e2bc1b5427644dc2f.jpg temp/1748386168_2924570_1360988016_f8d7a2fc6160b01f305553a575b3e058.jpg temp/1748386168_2924570_1360988014_0eeb78fb7fcc1610cad8a8e7387ed435.jpg temp/1748386168_2924570_1360988013_11f50a69571a47300f9012581b63c68b.jpg temp/1748386168_2924570_1360988012_66b1d0afab8d08da5b35a81689579d0d.jpg temp/1748386168_2924570_1360988009_66f42115a236a6bdeea528adb34f1315.jpg temp/1748386168_2924570_1360988006_cd788a2028a0e478a67039b9500fe7ec.jpg temp/1748386168_2924570_1360988004_69ca2bcec4193adcdd1b438b75924f03.jpg temp/1748386168_2924570_1360988003_9339dacb94e3a81a2e8b9cf92c34748a.jpg temp/1748386168_2924570_1360988002_b7b6a95a562c0adb8a32c2e8b66435c4.jpg temp/1748386168_2924570_1360988001_4c5013bbeae0ea924198707814e0e21a.jpg temp/1748386168_2924570_1360987988_43ba846acbf1d976d7e108dcb3641806.jpg temp/1748386168_2924570_1360987984_caff1637467640739ad50d8fad43eb79.jpg temp/1748386168_2924570_1360987982_c86e156736e1ecd7be73be93ac608fac.jpg temp/1748386168_2924570_1360987981_168710527190d9af88295b82d282f975.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1360988430, 1360988427, 1360988425, 1360988423, 1360988420, 1360988418, 1360988397, 1360988395, 1360988394, 1360988393, 1360988391, 1360988389, 1360988370, 1360988367, 1360988365, 1360988364, 1360988363, 1360988362, 1360988331, 1360988329, 1360988328, 1360988327, 1360988325, 1360988323, 1360988310, 1360988307, 1360988305, 1360988304, 1360988303, 1360988301, 1360988284, 1360988280, 1360988277, 1360988274, 1360988270, 1360988267, 1360988248, 1360988246, 1360988243, 1360988241, 1360988239, 1360988237, 1360988235, 1360988233, 1360988231, 1360988230, 1360988229, 1360988227, 1360988212, 1360988210, 1360988209, 1360988208, 1360988206, 1360988203, 1360988145, 1360988142, 1360988139, 1360988137, 1360988136, 1360988133, 1360988129, 1360988127, 1360988126, 1360988125, 1360988123, 1360988121, 1360988102, 1360988100, 1360988099, 1360988098, 1360988096, 1360988094, 1360988080, 1360988078, 1360988077, 1360988076, 1360988075, 1360988074, 1360988060, 1360988057, 1360988055, 1360988053, 1360988051, 1360988049, 1360988022, 1360988019, 1360988016, 1360988014, 1360988013, 1360988012, 1360988009, 1360988006, 1360988004, 1360988003, 1360988002, 1360988001, 1360987988, 1360987984, 1360987982, 1360987981] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.06446433067321777 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 13.27486801147461 time spend to save output : 0.06923484802246094 total time spend for step 0 : 13.34410285949707 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 12.17user 4.63system 0:36.25elapsed 46%CPU (0avgtext+0avgdata 2482612maxresident)k 1040inputs+164680outputs (0major+767320minor)pagefaults 0swaps