python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1550892 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 17815>100 please execute split_portfolio.py -i 22792232 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2977394'] with mtr_portfolio_ids : ['22792232'] and first list_photo_ids : [] new path : /proc/1550892/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 18.539624452590942 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu May 29 09:37:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-29 09:37:52.364570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-29 09:37:52.524201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 09:37:52.526690: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 09:37:52.568066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 09:37:52.591161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 09:37:52.597318: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 09:37:52.636040: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 09:37:52.650846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 09:37:52.714359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 09:37:52.728425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 09:37:52.728839: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-29 09:37:52.771164: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-29 09:37:52.773432: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f85f4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-29 09:37:52.773465: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-29 09:37:53.166777: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1b687480 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-29 09:37:53.166828: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-29 09:37:53.167779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 09:37:53.167891: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 09:37:53.167925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 09:37:53.167957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 09:37:53.167987: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 09:37:53.168015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 09:37:53.168046: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 09:37:53.168079: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 09:37:53.169777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 09:37:53.170465: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 09:37:53.171547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-29 09:37:53.171569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-29 09:37:53.171582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-29 09:37:53.172741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3449 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-29 09:38:00.929383: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 11.40239143371582 time used to load_weights : 0.6070461273193359 0it [00:00, ?it/s] 25it [00:00, 246.49it/s] 57it [00:00, 288.28it/s] 91it [00:00, 310.24it/s] 100it [00:00, 285.68it/s]2025-05-29 09:38:08.048318: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 09:38:08.501713: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-29 09:38:08.545132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 09:38:08.568766: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-29 09:38:08.572164: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748504247_1550892_1361511151_3b505e8bef585adddaebca76d12b0347_22719512.jpg temp/1748504247_1550892_1361511144_7f025a10b2c4665d727484fe7d933a5b_34818273.jpg temp/1748504247_1550892_1361511143_760fdcd45273cf82b5f1772e2690122c_87360273.jpg temp/1748504247_1550892_1361511142_e881becc6f02a640f8eb276b18d8dde0_58916893.jpg temp/1748504247_1550892_1361511141_7d7f6633902af84c624124d6f7b74b80_99575366.jpg temp/1748504247_1550892_1361511140_f073040be0accf5fb3c8eb43e8ea2260_82099559.jpg temp/1748504247_1550892_1361511139_c50de01698e82924b639c3f830aa6c00_47319647.jpg temp/1748504247_1550892_1361511118_b99701ba03a05a1b237ff373120f33fc_6328762.jpg temp/1748504247_1550892_1361511117_6990dd708447e352aa53599f517affdb_9063837.jpg temp/1748504247_1550892_1361511116_d5582dde7641c06f7f080762503f0f2a_72120613.jpg temp/1748504247_1550892_1361511115_d00188bc1157cc7d86ad8b69c26e3e72_70716471.jpg temp/1748504247_1550892_1361511114_b793deee66ddaf628f7ce0760bd2cac1_92132237.jpg temp/1748504247_1550892_1361511113_90e9fbe5aec6005a7f895ec24d0c71eb_25784473.jpg temp/1748504247_1550892_1361511105_1796078afb4d299949b024f8ddbb2469_60816135.jpg temp/1748504247_1550892_1361511104_a5bb6964a6dfad2995d3a115f71a9d02_28350243.jpg temp/1748504247_1550892_1361511103_3351edd26a9a789ba15b9ea8b89843ef_97150651.jpg temp/1748504247_1550892_1361511102_308a4fc0285d0e0a53e3d12f51189610_96182750.jpg temp/1748504247_1550892_1361511101_5efbe27da381a70f9ace9326db2ae3df_14263180.jpg temp/1748504247_1550892_1361511100_49968b33a08031d1fb3537d7c01999fe_69474559.jpg temp/1748504247_1550892_1361511085_d96adc52c2109dc5338ee13b50172ea6_90236856.jpg temp/1748504247_1550892_1361511084_ce52e3ae7b14d03e06272076a9a8844f_38835287.jpg temp/1748504247_1550892_1361511083_e76f28a88004e14e4b543bd46d63fba5_26057712.jpg temp/1748504247_1550892_1361511082_b95cdb07f77ac880af49c44b33afa6f7_90117276.jpg temp/1748504247_1550892_1361511081_db4806b57170baa70a7f8bab2f2c9157_54176266.jpg temp/1748504247_1550892_1361511080_9c378d5a1c3c2c35b4f4ea91f185999f_18497865.jpg temp/1748504247_1550892_1361511076_58eb118f8b2d8578e57e9f1d20ebc98e_31414598.jpg temp/1748504247_1550892_1361511074_6b67d0fe57ee7dba35e91037543302c5_22193567.jpg temp/1748504247_1550892_1361511071_7a503467edaffc91dd955ec6f492a01a_24544104.jpg temp/1748504247_1550892_1361511069_4b4d7bc2cfea51008ac9553872dd58ff_16570871.jpg temp/1748504247_1550892_1361511067_377f9105b10cd9a0916363c6fc20dc19_19186204.jpg temp/1748504247_1550892_1361511058_e3f631fe57a17470e569193e2e0b96b9_18621482.jpg temp/1748504247_1550892_1361511057_eb90d2d012ee9e252e42f5c6c4ffeabf_99026530.jpg temp/1748504247_1550892_1361511056_307e5fdfb40af52b1ee22145cca01ae7_60147164.jpg temp/1748504247_1550892_1361511055_743cb649b35074576f8bfff06edb9124_51688680.jpg temp/1748504247_1550892_1361511054_16e03a21d0be0b3c0e164db23980fcca_96590817.jpg temp/1748504247_1550892_1361511053_a476657e39e65063ab2e1e9b5e9f7345_70844001.jpg temp/1748504247_1550892_1361511052_448892ecb1de1dc247fbe8327388471d_51134912.jpg temp/1748504247_1550892_1361511051_246cb1ac2048fede51cc416c47887647_55836774.jpg temp/1748504247_1550892_1361511050_1809fc2037d13efb0d0d839e9b7d0a27_76810499.jpg temp/1748504247_1550892_1361511049_68afc71d02184e8322f343e90744a35b_81007550.jpg temp/1748504247_1550892_1361511048_277deafdcff948938ce4c48e34979bb2_89322920.jpg temp/1748504247_1550892_1361511047_d31d9751b22d97032e6ec742b4ad1e9e_23638182.jpg temp/1748504247_1550892_1361511037_485d8ddcd8901793da76a47f926e4814_74071938.jpg temp/1748504247_1550892_1361511035_c4b9b7aa2f6f1c528d27b6f480f7a1f3_75030789.jpg temp/1748504247_1550892_1361511034_4ea15361408b7739d6d9caad05b617ad_1736166.jpg temp/1748504247_1550892_1361511033_468b1dbc3240705b9447ada4ce41729f_93799563.jpg temp/1748504247_1550892_1361511032_1a06562a619baeeb124cc853313ccc33_72212311.jpg temp/1748504247_1550892_1361511031_6eaf1bc3330602253686a25376e1eb13_91501430.jpg temp/1748504247_1550892_1361511030_e9bf87f1837e5182d2489b44eb1683c8_38166070.jpg temp/1748504247_1550892_1361511029_12da8f4b28dd4cf8d98b33865eead4e1_75967921.jpg temp/1748504247_1550892_1361511028_82d5a03dc36b2c53f62a731742e5bbbf_2312246.jpg temp/1748504247_1550892_1361511027_095ade814c7cea52d20fda42f0df5f97_89955608.jpg temp/1748504247_1550892_1361511026_687302597e507e3a3254ee6299ccf2b4_74295453.jpg temp/1748504247_1550892_1361511025_5e8997100e01ded27964f530f4d15952_12581116.jpg temp/1748504247_1550892_1361511010_a6c13213df07b4a40ba13ebe67240fab_36325897.jpg temp/1748504247_1550892_1361511009_49a7aedca620d040446bd1c231041b88_4149409.jpg temp/1748504247_1550892_1361511008_da136df5ee40fa032e6d5cbc196ab1df_59250993.jpg temp/1748504247_1550892_1361511007_db2837a5e43c615a5a46bb6bae5fb136_88413390.jpg temp/1748504247_1550892_1361511005_66cee24d48deeb9c79d3d3dc675e4d10_23729976.jpg temp/1748504247_1550892_1361511002_fd0af3725f44edee7f4e0c33ae6f1878_65168830.jpg temp/1748504247_1550892_1361510995_9b183d5d5b395c2bc0b9f90db3765473_7077729.jpg temp/1748504247_1550892_1361510994_c33752c8fc3e550f710106b69bd5d40c_67706609.jpg temp/1748504247_1550892_1361510993_5a1810fd318fd923b1698921af9e65ca_37999795.jpg temp/1748504247_1550892_1361510992_07603848ed2bd64f0f03abf8ab836b9c_13355403.jpg temp/1748504247_1550892_1361510991_16ede3ba56e036bbe772709ce4b5bd92_41151268.jpg temp/1748504247_1550892_1361510990_0faad4fb5708050dab1e5ff6c53ecfc9_37122634.jpg temp/1748504247_1550892_1361510981_98f7e6437000cb6e29a9665eb5f855c6_98424376.jpg temp/1748504247_1550892_1361510979_79cec43d3b488eb558a9f371cc0c1f69_34867009.jpg temp/1748504247_1550892_1361510978_b3c997696b7fda9be8e362b1ce847fde_56119808.jpg temp/1748504247_1550892_1361510977_1cb94decc96e36d870ca2b865635b034_1450424.jpg temp/1748504247_1550892_1361510976_7ed8323078bd8a05ba15c2226383c15d_80506741.jpg temp/1748504247_1550892_1361510975_fa57faf44f01000c2048e2615098690e_11154274.jpg temp/1748504247_1550892_1361510965_58d503345e00a6e4a2ae64003feecf80_57591044.jpg temp/1748504247_1550892_1361510964_3e67b115ca5d03891c21508a423e201e_23445407.jpg temp/1748504247_1550892_1361510963_349ddf83e12efe194c2b4a3385859374_79475696.jpg temp/1748504247_1550892_1361510962_f07174f9f32fdad759a8888366280326_82755967.jpg temp/1748504247_1550892_1361510961_10476e74df9d26567a719d902b869ed1_3170458.jpg temp/1748504247_1550892_1361510960_56792442a05a422184258c94f27fcbe0_30882773.jpg temp/1748504247_1550892_1361510958_dfd1a95de7c4dfbc2652bb5bb34ebf57_74739636.jpg temp/1748504247_1550892_1361510949_de45441b8b73aa8c911be5a5df97b5c8_30953846.jpg temp/1748504247_1550892_1361510947_be7a4040a24236d9dd916fcf74bc8daa_28046325.jpg temp/1748504247_1550892_1361510946_eeb5fb982a60a6471af87cc1c0842c85_98309938.jpg temp/1748504247_1550892_1361510945_feb98dffbdd69e5a52e140ab60fe3a7a_49781377.jpg temp/1748504247_1550892_1361510944_5f83622f4093457be1fb973756a559d8_72157401.jpg temp/1748504247_1550892_1361510943_dda4c7f8d7121ad813f98e26d8643443_68462864.jpg temp/1748504247_1550892_1361510942_bde29dc0694ef48277431c0f142c0e53_68367480.jpg temp/1748504247_1550892_1361510941_0b0231bf31b5b1caab7f4d5807e2ba0a_60737483.jpg temp/1748504247_1550892_1361510940_da10593653f01fedc874732d14885d62_80852704.jpg temp/1748504247_1550892_1361510939_5f92e5dc8c6400717bba9ec83062acf3_97194406.jpg temp/1748504247_1550892_1361510930_bcc1a938d12f4ff35a289665db5bbcd2_22370606.jpg temp/1748504247_1550892_1361510929_ab655d3ccbb0e71bae30c35da12c4238_97302380.jpg temp/1748504247_1550892_1361510928_a708088e52ab715c3d578e412c8b1196_12610196.jpg temp/1748504247_1550892_1361510927_45a19b3a2860554b2dfbcd11baa52795_20972615.jpg temp/1748504247_1550892_1361510926_4db81dc12bd3f15c466a1a6ede39d50a_85122626.jpg temp/1748504247_1550892_1361510924_e26ed007f517387a6b0b490b5ac941f6_3657075.jpg temp/1748504247_1550892_1361510917_4bc55ea8d697697bf2740bc8753b07b4_83121839.jpg temp/1748504247_1550892_1361510915_b6c65a2cf234a3ef543d56ffcfd7b0db_78776442.jpg temp/1748504247_1550892_1361510913_56e4ad88e9913bd60511f8a45eaf0e71_38076578.jpg temp/1748504247_1550892_1361510911_0f2ad7aa937af02b0e9629a27ff72bdb_14642885.jpg temp/1748504247_1550892_1361510909_3056e5144c804803c1d89990e764a5b4_1942738.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361511151, 1361511144, 1361511143, 1361511142, 1361511141, 1361511140, 1361511139, 1361511118, 1361511117, 1361511116, 1361511115, 1361511114, 1361511113, 1361511105, 1361511104, 1361511103, 1361511102, 1361511101, 1361511100, 1361511085, 1361511084, 1361511083, 1361511082, 1361511081, 1361511080, 1361511076, 1361511074, 1361511071, 1361511069, 1361511067, 1361511058, 1361511057, 1361511056, 1361511055, 1361511054, 1361511053, 1361511052, 1361511051, 1361511050, 1361511049, 1361511048, 1361511047, 1361511037, 1361511035, 1361511034, 1361511033, 1361511032, 1361511031, 1361511030, 1361511029, 1361511028, 1361511027, 1361511026, 1361511025, 1361511010, 1361511009, 1361511008, 1361511007, 1361511005, 1361511002, 1361510995, 1361510994, 1361510993, 1361510992, 1361510991, 1361510990, 1361510981, 1361510979, 1361510978, 1361510977, 1361510976, 1361510975, 1361510965, 1361510964, 1361510963, 1361510962, 1361510961, 1361510960, 1361510958, 1361510949, 1361510947, 1361510946, 1361510945, 1361510944, 1361510943, 1361510942, 1361510941, 1361510940, 1361510939, 1361510930, 1361510929, 1361510928, 1361510927, 1361510926, 1361510924, 1361510917, 1361510915, 1361510913, 1361510911, 1361510909] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.5943043231964111 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 22.386202335357666 time spend to save output : 0.60518479347229 total time spend for step 0 : 22.991387128829956 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.93user 8.02system 0:46.78elapsed 46%CPU (0avgtext+0avgdata 1649792maxresident)k 2132688inputs+178952outputs (8188major+621234minor)pagefaults 0swaps