python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2408084 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2028>100 please execute split_portfolio.py -i 23034673 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2869812'] with mtr_portfolio_ids : ['23034673'] and first list_photo_ids : [] new path : /proc/2408084/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 13.080700159072876 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri May 16 06:31:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-05-16 06:31:49.470884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-16 06:31:49.500269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 06:31:49.500536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 06:31:49.502565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 06:31:49.504801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 06:31:49.505142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 06:31:49.507526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 06:31:49.508653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 06:31:49.513411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 06:31:49.514515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 06:31:49.515022: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-16 06:31:49.539123: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-16 06:31:49.540668: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f80d4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-16 06:31:49.540721: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-16 06:31:49.775904: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1cd8d2e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-16 06:31:49.775944: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-16 06:31:49.776756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 06:31:49.776839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 06:31:49.776856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 06:31:49.776872: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 06:31:49.776887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 06:31:49.776902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 06:31:49.776917: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 06:31:49.776932: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 06:31:49.777689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 06:31:49.777735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 06:31:49.778260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-16 06:31:49.778274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-16 06:31:49.778282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-16 06:31:49.779106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2962 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-16 06:31:56.897479: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-16 06:31:56.898085: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.513280868530273 time used to load_weights : 0.26366114616394043 0it [00:00, ?it/s] 100it [00:00, 1601.85it/s]2025-05-16 06:32:00.685662: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 06:32:01.044855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 06:32:01.345976: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-16 06:32:01.347992: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1747369888_2408084_1358825795_2fcee98067f666af1c24537249699be1.jpg temp/1747369888_2408084_1358825712_1c2528526624a318ed0e63bdbadda497.jpg temp/1747369888_2408084_1358825674_e30c5941aa1bb742ab7296280443f578.jpg temp/1747369888_2408084_1358825643_bceb8189c1f4c03dc56a709bbaf52516.jpg temp/1747369888_2408084_1358825633_5d12049835ee2cbe69332d1a9300546b.jpg temp/1747369888_2408084_1358825631_2dbfaba95749a4ed7dd90f3455ff11d6.jpg temp/1747369888_2408084_1358825159_17e02c0948b99173603a7af5e40082f7.jpg temp/1747369888_2408084_1358825025_ca1b8fce14373ea28f3ccdf3d71010e5.jpg temp/1747369888_2408084_1358824897_6e012ad61165fa82e6ce4222c53f8f03.jpg temp/1747369888_2408084_1358824789_8f1542c6bbce92bc56ca00fa8b1c07d5.jpg temp/1747369888_2408084_1358824684_457214261c7c377e56993c5287b802c5.jpg temp/1747369888_2408084_1358824583_f8bcad7781f49e4aa6b7f865a8fd686a.jpg temp/1747369888_2408084_1358824236_0d379e215748b095c9c97dc723176a0b.jpg temp/1747369888_2408084_1358824211_e71d1b4f2c4962f3238da309f3c61901.jpg temp/1747369888_2408084_1358824181_22442eb9a4cc82271b3838fbf69ddbf8.jpg temp/1747369888_2408084_1358824157_578bda75e5cd6e93ab61dbe1eec8f5fe.jpg temp/1747369888_2408084_1358824132_99ee3f820d5993a73047cb6f06bf713e.jpg temp/1747369888_2408084_1358824105_1a6ec6387d98804ecb6bd84ca62805f6.jpg temp/1747369888_2408084_1358823837_671c4a12b4406a72e3b97409f334e859.jpg temp/1747369888_2408084_1358823807_4659837d33abba4fd5fa86f149b0a15d.jpg temp/1747369888_2408084_1358823778_fdb67938edc6a9d62e8a17e7cdb59880.jpg temp/1747369888_2408084_1358823769_07bb06da549c0a042f8079a039e80abc.jpg temp/1747369888_2408084_1358823768_0a9c9d27ebfd25d07d6357c382dcc21a.jpg temp/1747369888_2408084_1358823766_b36770969d845709db2ccc7c36f08d71.jpg temp/1747369888_2408084_1358823718_240c69c5919883b82b60c0c262620758.jpg temp/1747369888_2408084_1358823709_984190eac23d0f6ea015dd83c4c89c33.jpg temp/1747369888_2408084_1358823694_7dcca2ab6842d502bf7ba0deb995c410.jpg temp/1747369888_2408084_1358823668_8ea776d01e7effefd50e92a8de7c3c6b.jpg temp/1747369888_2408084_1358823653_d6dd2be94db2b4cb8cb15fe322e74d37.jpg temp/1747369888_2408084_1358823643_f73904d3c82b1d6696338ae7828f8b54.jpg temp/1747369888_2408084_1358823350_bed9f9475b9d09df6cb243b6110280f1.jpg temp/1747369888_2408084_1358823295_c913c82d132999a71a8743c65c487396.jpg temp/1747369888_2408084_1358823263_d866daf432f2fc9dd3fb13bbe47f40d1.jpg temp/1747369888_2408084_1358823236_64910f83925d065d5ada31a083288ef7.jpg temp/1747369888_2408084_1358823223_fa32d5c80758bdb8a8138dd00440671f.jpg temp/1747369888_2408084_1358823220_65bad7e9221e8dbbe14004d38fa9eaa3.jpg temp/1747369888_2408084_1358822969_57e5f4777175483c9ad1c5bdb163bc7d.jpg temp/1747369888_2408084_1358822942_2747164b75a4a12e92a4992d9d326dad.jpg temp/1747369888_2408084_1358822915_161989f6e32f9fd0e1357dd3ef0de64e.jpg temp/1747369888_2408084_1358822888_4c95bfe47491830f1b08ec6c58bcb5c2.jpg temp/1747369888_2408084_1358822858_6343ca7eb64c0bc70cd4b10189bf4f23.jpg temp/1747369888_2408084_1358822788_d7792faff8f5b8d8ea73efd7b44fd6f4.jpg temp/1747369888_2408084_1358821826_697953563b08e9478d7c74432b367b37.jpg temp/1747369888_2408084_1358821800_945d662c796bd8e5ae5ec0b655ed9af5.jpg temp/1747369888_2408084_1358821650_459702de6ea87d5d59a1643c5bf98b4e.jpg temp/1747369888_2408084_1358821536_f5179bf6341c5dabbbe7cbe6c51acd06.jpg temp/1747369888_2408084_1358821361_625e78170775570d6280156765473c34.jpg temp/1747369888_2408084_1358821279_bc51b694ceff1de86f5a13178d8279c7.jpg temp/1747369888_2408084_1358820454_175467ec417ead8370a3869ab7106ff6.jpg temp/1747369888_2408084_1358820427_d1e970f0cb04a5b8091be1d938577c0c.jpg temp/1747369888_2408084_1358820399_4a204080d54db5e7d59e103a6f0bcff9.jpg temp/1747369888_2408084_1358820373_205a2233c92f3837d15a4c6f161df437.jpg temp/1747369888_2408084_1358820345_737612e3662968892e1145d2fd551cfb.jpg temp/1747369888_2408084_1358820318_b17bd017c7e4c9e3c2cd321835c59b53.jpg temp/1747369888_2408084_1358820279_37c7b55993aa6e97fba929dd2610504c.jpg temp/1747369888_2408084_1358820257_2d657d214175b1c3c58a59b2ac95ff8a.jpg temp/1747369888_2408084_1358820231_e6b78fc64decf47e0640c37071b00527.jpg temp/1747369888_2408084_1358820205_1f2c09f240ac7807587762e52dfbe956.jpg temp/1747369888_2408084_1358820163_adcc832f6230d3dc41335ce132e351e4.jpg temp/1747369888_2408084_1358820134_a73b74d3347e7167ff44bd632838daef.jpg temp/1747369888_2408084_1358819717_c4315fec76697b55c8f02ba14fc0d89e.jpg temp/1747369888_2408084_1358819663_562036f6d000450b796d3f440a5ce78a.jpg temp/1747369888_2408084_1358819635_837235c248980111d2b1e09ac832fe4b.jpg temp/1747369888_2408084_1358819606_380e85a5e7e57707a752ce8fc1a1aabb.jpg temp/1747369888_2408084_1358819540_09f3ba0929dbe02ab03f4c7c3ab07945.jpg temp/1747369888_2408084_1358819465_8ab5fd4839b4462218233fad813ecfd6.jpg temp/1747369888_2408084_1358818911_0792500057a84688212cc521131ef42b.jpg temp/1747369888_2408084_1358818856_db86e606d0757fce9117ea9825b50355.jpg temp/1747369888_2408084_1358818805_a3a02e3f83f21f8de1591a4cc09757dc.jpg temp/1747369888_2408084_1358818752_5bd76b5979bdfee2d4865b0f31771743.jpg temp/1747369888_2408084_1358818700_de22d80baae0475718e8b34c0d7668fa.jpg temp/1747369888_2408084_1358818646_cb7de66435188dcd7ec5fc0be9a4e0ca.jpg temp/1747369888_2408084_1358818164_ed2994ed320e2b668b12aa702436e190.jpg temp/1747369888_2408084_1358818139_01861da99e58636700cebb19b8c7e563.jpg temp/1747369888_2408084_1358818108_d32c5e0c93e15a930701a6b0331f2449.jpg temp/1747369888_2408084_1358818080_d5942ccb3b7fcfa73c1c789c4d54dc3c.jpg temp/1747369888_2408084_1358818056_44ef335fe7d6d7f354de2944f7eaa5a8.jpg temp/1747369888_2408084_1358818029_94928aae3a2cc7f0ec28e927719fb9ab.jpg temp/1747369888_2408084_1358817662_8201329034644f3db852fc3afd6665de.jpg temp/1747369888_2408084_1358817630_2513ae836f5cd2cf2a9add8dd7c35084.jpg temp/1747369888_2408084_1358817579_8a2921fa6e40f4bf7e479c7ea56a983b.jpg temp/1747369888_2408084_1358817527_abe0a892df51ae500ab864679bf59ec7.jpg temp/1747369888_2408084_1358817474_5fc6ed6f19a9c12d0bb007729e210885.jpg temp/1747369888_2408084_1358817421_cec3aebc6e03e052cbf02c8ab943321c.jpg temp/1747369888_2408084_1358817091_cf9c855ef29e458f4a2deafc737c4e1b.jpg temp/1747369888_2408084_1358817062_a804d1b1b9a023a48ce53919033f7270.jpg temp/1747369888_2408084_1358817035_9f89faf9eee40e6c0978ab34bc0a02b2.jpg temp/1747369888_2408084_1358817008_92aa4e03c5bdf7cc586a554d00fa4501.jpg temp/1747369888_2408084_1358816932_4cf202b37e5e0644b424e03f42b7da09.jpg temp/1747369888_2408084_1358816878_450564e5bdbe785c402866f50a102d54.jpg temp/1747369888_2408084_1358816476_c289cf40ec3a47595bb9309081b326cf.jpg temp/1747369888_2408084_1358816446_a8c48ab867545974fa8559c23314a46a.jpg temp/1747369888_2408084_1358816416_bfe1ca94207949699a0d1b3a3fbcbd07.jpg temp/1747369888_2408084_1358816319_df63298eac434baabc0e62fa2ed302c6.jpg temp/1747369888_2408084_1358816281_8e83e93838bd656c33dd8f16f364af8e.jpg temp/1747369888_2408084_1358816233_f15133e57c69449766876393aa7fa38a.jpg temp/1747369888_2408084_1358814584_60b57a67e7ab0537a556a2df26fd8a4f.jpg temp/1747369888_2408084_1358814553_ab19948e5831a259fbddcf2d766395b5.jpg temp/1747369888_2408084_1358814543_bcfc9f262accb1dcbb9679e294a7fecf.jpg temp/1747369888_2408084_1358814534_199fa7ebae29e3365e28d753a0ac7794.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1358825795, 1358825712, 1358825674, 1358825643, 1358825633, 1358825631, 1358825159, 1358825025, 1358824897, 1358824789, 1358824684, 1358824583, 1358824236, 1358824211, 1358824181, 1358824157, 1358824132, 1358824105, 1358823837, 1358823807, 1358823778, 1358823769, 1358823768, 1358823766, 1358823718, 1358823709, 1358823694, 1358823668, 1358823653, 1358823643, 1358823350, 1358823295, 1358823263, 1358823236, 1358823223, 1358823220, 1358822969, 1358822942, 1358822915, 1358822888, 1358822858, 1358822788, 1358821826, 1358821800, 1358821650, 1358821536, 1358821361, 1358821279, 1358820454, 1358820427, 1358820399, 1358820373, 1358820345, 1358820318, 1358820279, 1358820257, 1358820231, 1358820205, 1358820163, 1358820134, 1358819717, 1358819663, 1358819635, 1358819606, 1358819540, 1358819465, 1358818911, 1358818856, 1358818805, 1358818752, 1358818700, 1358818646, 1358818164, 1358818139, 1358818108, 1358818080, 1358818056, 1358818029, 1358817662, 1358817630, 1358817579, 1358817527, 1358817474, 1358817421, 1358817091, 1358817062, 1358817035, 1358817008, 1358816932, 1358816878, 1358816476, 1358816446, 1358816416, 1358816319, 1358816281, 1358816233, 1358814584, 1358814553, 1358814543, 1358814534] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.06417727470397949 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 19.813324689865112 time spend to save output : 0.07184505462646484 total time spend for step 0 : 19.885169744491577 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.53user 4.41system 0:37.53elapsed 47%CPU (0avgtext+0avgdata 2059488maxresident)k 443136inputs+161160outputs (246major+637875minor)pagefaults 0swaps