python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1009063 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 3150>100 please execute split_portfolio.py -i 22406937 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2759767'] with mtr_portfolio_ids : ['22406937'] and first list_photo_ids : [] new path : /proc/1009063/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 12.438812732696533 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri Apr 25 10:46:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2277 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2277 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2277 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2277 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2277 wait 20 seconds inside check gpu memory 2025-04-25 10:48:31.509419: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-04-25 10:48:31.556920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-25 10:48:31.559811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-25 10:48:31.600345: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-25 10:48:31.622893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-25 10:48:31.627421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-25 10:48:31.666215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-25 10:48:31.673297: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-25 10:48:31.741965: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-25 10:48:31.743410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-25 10:48:31.744848: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-04-25 10:48:31.783380: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-04-25 10:48:31.785495: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f0578000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-04-25 10:48:31.785561: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-04-25 10:48:32.013457: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1fc07cb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-04-25 10:48:32.013493: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-04-25 10:48:32.014625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-25 10:48:32.014685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-25 10:48:32.014704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-25 10:48:32.014720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-25 10:48:32.014736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-25 10:48:32.014752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-25 10:48:32.014768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-25 10:48:32.014783: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-25 10:48:32.015651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-25 10:48:32.016024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-25 10:48:32.017446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-25 10:48:32.017462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-25 10:48:32.017470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-25 10:48:32.018378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3470 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.542036533355713 time used to load_weights : 0.5457251071929932 0it [00:00, ?it/s] 62it [00:00, 613.43it/s] 100it [00:00, 626.05it/s]2025-04-25 10:48:44.777666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-25 10:48:45.534145: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-25 10:48:45.721127: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-04-25 10:48:45.744337: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1745570788_1009063_1353665644_a61486cca3cf14f611d0495468862b19.jpg temp/1745570788_1009063_1353665642_6169090254777b104321a8ae43c1a3f2.jpg temp/1745570788_1009063_1353665638_4e98f92b0a6e69d176edc44b648fab34.jpg temp/1745570788_1009063_1353665634_2fcf75a856c4e8fba108c5c173b94a83.jpg temp/1745570788_1009063_1353665631_9d47bce4ec65e4fd730dcc6d1031f9ec.jpg temp/1745570788_1009063_1353665627_5f5f5b62306ad641452b84f5d70155fe.jpg temp/1745570788_1009063_1353665600_3c583a150fa251a84c775db8c9f432d1.jpg temp/1745570788_1009063_1353665595_9ae1b127d9e0e9b9bac0c383437774ae.jpg temp/1745570788_1009063_1353665591_a0ee1145b86f82c3d128a3cde6605d92.jpg temp/1745570788_1009063_1353665586_778ce5f3a8dd0b2ab188839218043e76.jpg temp/1745570788_1009063_1353665582_208f5d20c6c89f0563bd21ab6c54b803.jpg temp/1745570788_1009063_1353665577_fd5dae9a7cf35cc333733f97dff4c0b1.jpg temp/1745570788_1009063_1353665550_2b39e51d6b5de6f71b3317c7aeb1aadf.jpg temp/1745570788_1009063_1353665534_7cd813e38ae2d6876a3c57c019c565b1.jpg temp/1745570788_1009063_1353665502_b82ae3b4bd08b197f80c1b6d31f52818.jpg temp/1745570788_1009063_1353665471_331ca2fbde2b0723ed885d0cf0ebfe11.jpg temp/1745570788_1009063_1353665437_1f08e51777e1e7e67832322c6bdecbb9.jpg temp/1745570788_1009063_1353665408_62ee8c76d8c6adf1195a2037155b4c0b.jpg temp/1745570788_1009063_1353665316_9954d5b30d301cba8491ea0a516d8c1c.jpg temp/1745570788_1009063_1353665314_3beac9bf036a705d0ea15d420471ba97.jpg temp/1745570788_1009063_1353665307_97d749c8b0ba963ac3fd33542874a6c5.jpg temp/1745570788_1009063_1353665270_5b48d9ed3ef94683fa3c1e9233791734.jpg temp/1745570788_1009063_1353665222_45f858c32acf9c8244e08f4a5d994072.jpg temp/1745570788_1009063_1353665208_112d0ba1434205990ed836c0d6d767cc.jpg temp/1745570788_1009063_1353665011_000c65a0c131b09cc6e5b3ee8298782e.jpg temp/1745570788_1009063_1353665002_11c99bbfae0396ae6e2d92c017ce4ece.jpg temp/1745570788_1009063_1353664897_17ff98c9286c463e376e03ba44775a04.jpg temp/1745570788_1009063_1353664891_0e76e432904340d0605404e2925875f4.jpg temp/1745570788_1009063_1353664887_86d6f384d7d534b2be294ab28612b7c2.jpg temp/1745570788_1009063_1353664884_d3ecce3db67fcc5eb5806418966c28fa.jpg temp/1745570788_1009063_1353664022_573bd0ad8c25d814bb6612a3f13a874f.jpg temp/1745570788_1009063_1353664019_6a8881f4ffcf5ae5da6ffb9667a38c3a.jpg temp/1745570788_1009063_1353664015_1527c07ea7065f05af6fae3f7b705daf.jpg temp/1745570788_1009063_1353664011_011254d1a012206273891b8eaa034ea3.jpg temp/1745570788_1009063_1353664008_daf0424e8224fd07211da765530e44fb.jpg temp/1745570788_1009063_1353664006_ea973ade4224a5b972afd9963f21eed9.jpg temp/1745570788_1009063_1353663735_e85bb69bcca9da2ea7c92283fe88c74b.jpg temp/1745570788_1009063_1353663709_cc18159d81797ea01750d600da914a1d.jpg temp/1745570788_1009063_1353663681_14d7dc6b952dab806c555a5dac007f37.jpg temp/1745570788_1009063_1353663655_c2f06dad871de72c97388dd148f8fb52.jpg temp/1745570788_1009063_1353663646_ca9dd3e661d8845685ef64adc804ae18.jpg temp/1745570788_1009063_1353663635_7907bdb78e32e861771d1a6fd187208e.jpg temp/1745570788_1009063_1353663489_1883a71f033a32fc62e21211114abdba.jpg temp/1745570788_1009063_1353663334_6b2f369c3cd5b1f2dbe042b627c15ff6.jpg temp/1745570788_1009063_1353663330_8cdcb6e6c7d8fe8dec81d449c8d46995.jpg temp/1745570788_1009063_1353663326_63cb2fa5d8e2faa51efc6a6bb3c18a29.jpg temp/1745570788_1009063_1353663323_5c3482a070f92033cb7d2d20bfc4675c.jpg temp/1745570788_1009063_1353663317_030b806a0b846149c39caa0b913d6e90.jpg temp/1745570788_1009063_1353663275_f1f0f5f9195cd677c22392885c4170b8.jpg temp/1745570788_1009063_1353663272_7303dcd85abc25d2508ffa36b6702c69.jpg temp/1745570788_1009063_1353663268_f8d6033e9634b75f65edb37e8e879bdc.jpg temp/1745570788_1009063_1353663265_d0275f14bac0c249fbcd5ba88729595b.jpg temp/1745570788_1009063_1353663262_5470bf787d25534631377828ec5f284a.jpg temp/1745570788_1009063_1353663259_bc96596e0506a1bb69da04e708e32986.jpg temp/1745570788_1009063_1353663196_49a9018d486c5d64c4a2f1f3f791f8d2.jpg temp/1745570788_1009063_1353663194_ec5869664f9ff36b9671169cb89f9879.jpg temp/1745570788_1009063_1353663190_df599a8e9e50032cfa70b064f05559d7.jpg temp/1745570788_1009063_1353663185_f77de67971f24e3bc64b107ae7a5dfaf.jpg temp/1745570788_1009063_1353663179_7a56c2191bb463c9666ecf30a463dd1b.jpg temp/1745570788_1009063_1353663168_63f3318f9db2e3918e054d8f45595a72.jpg temp/1745570788_1009063_1353663141_2342c851b4f9034aba80f8594d610db3.jpg temp/1745570788_1009063_1353663139_29858459e7913f1ea1a1f2dba48d7713.jpg temp/1745570788_1009063_1353663135_f8605200b6ab0f622d683f3ef036e0b1.jpg temp/1745570788_1009063_1353663131_32b8243efa362e1619e269e2a53ec7b6.jpg temp/1745570788_1009063_1353663126_a8651ea82bd22ca9255ca3998ef55598.jpg temp/1745570788_1009063_1353663121_596eadfe548f493961b266893841959b.jpg temp/1745570788_1009063_1353663064_5b21c69668465d38114482e3b16365fc.jpg temp/1745570788_1009063_1353663061_cd68bd7547fe92194ea9cdd71ac77ec4.jpg temp/1745570788_1009063_1353663058_996db5fd10d574a678c9645c8f6cb3a0.jpg temp/1745570788_1009063_1353663054_c5b3b74f5d6f33d7bca8ed433cc6518a.jpg temp/1745570788_1009063_1353663050_e06d8c18008261bdc45b46f486374e1c.jpg temp/1745570788_1009063_1353663045_cf7432b7d7b0c87137a977f5af831c96.jpg temp/1745570788_1009063_1353663010_f382316933ab767e9336ccd21b5e976e.jpg temp/1745570788_1009063_1353663009_77a1042dbeff826569bdaceb398fb201.jpg temp/1745570788_1009063_1353663006_ea49a7c52b975cc18359ea5a20ba5ff2.jpg temp/1745570788_1009063_1353663003_b681c6774ab40194e178852cad3197cf.jpg temp/1745570788_1009063_1353662999_5db585a8fe1c924c345d756a2c22af7d.jpg temp/1745570788_1009063_1353662994_0ccc8e641e9fdc070b92a641f6eeb8be.jpg temp/1745570788_1009063_1353662968_7c74599ea314e73f2570d7d71554a047.jpg temp/1745570788_1009063_1353662965_efad932a6f4b353eedb022927dca389e.jpg temp/1745570788_1009063_1353662954_a22049ba033483e2103ef2521d7daa5a.jpg temp/1745570788_1009063_1353662948_bd74d6f6f32a05adff004309956de83e.jpg temp/1745570788_1009063_1353662943_72c42be1d628530368fc2bdbdb5f0a6c.jpg temp/1745570788_1009063_1353662937_008519b320fe9a24ee5de67827e21729.jpg temp/1745570788_1009063_1353662912_874b218b0b2ad341689d3d36fb414c31.jpg temp/1745570788_1009063_1353662909_6fd2e71134421c702a208850f122cbdc.jpg temp/1745570788_1009063_1353662905_3243ebf43bc7b86658f9b77f8c18c1fe.jpg temp/1745570788_1009063_1353662902_99c6ed5b23f2acfa9db1d2362eb2ac87.jpg temp/1745570788_1009063_1353662899_613b20fb09896633c4e26364b138f9ae.jpg temp/1745570788_1009063_1353662894_f0f8f76cd8705077e59e88acd0b27547.jpg temp/1745570788_1009063_1353662815_190f9c38dcd380298ee86956926b90de.jpg temp/1745570788_1009063_1353662811_b3b3f3ace6d651020709988b19935e1a.jpg temp/1745570788_1009063_1353662806_257c66d7d3ae0ce35fdb8feaa047dde0.jpg temp/1745570788_1009063_1353662801_6692ed4d4d9a7dd9710a34ae511370b8.jpg temp/1745570788_1009063_1353662797_1cc52611a13d2c682cbae93a6bf6e54b.jpg temp/1745570788_1009063_1353662791_c1eb88cec3d142f709524192abe88e73.jpg temp/1745570788_1009063_1353662759_d88b6f504ee245184d077108e8100b88.jpg temp/1745570788_1009063_1353662757_cde3dbf5dde0bd21547fd5bc594c4698.jpg temp/1745570788_1009063_1353662750_8c74436c5933fe63cd5a8367c347ffa1.jpg temp/1745570788_1009063_1353662744_b99ef47b7cadf5671fa88e0c91681c35.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1353665644, 1353665642, 1353665638, 1353665634, 1353665631, 1353665627, 1353665600, 1353665595, 1353665591, 1353665586, 1353665582, 1353665577, 1353665550, 1353665534, 1353665502, 1353665471, 1353665437, 1353665408, 1353665316, 1353665314, 1353665307, 1353665270, 1353665222, 1353665208, 1353665011, 1353665002, 1353664897, 1353664891, 1353664887, 1353664884, 1353664022, 1353664019, 1353664015, 1353664011, 1353664008, 1353664006, 1353663735, 1353663709, 1353663681, 1353663655, 1353663646, 1353663635, 1353663489, 1353663334, 1353663330, 1353663326, 1353663323, 1353663317, 1353663275, 1353663272, 1353663268, 1353663265, 1353663262, 1353663259, 1353663196, 1353663194, 1353663190, 1353663185, 1353663179, 1353663168, 1353663141, 1353663139, 1353663135, 1353663131, 1353663126, 1353663121, 1353663064, 1353663061, 1353663058, 1353663054, 1353663050, 1353663045, 1353663010, 1353663009, 1353663006, 1353663003, 1353662999, 1353662994, 1353662968, 1353662965, 1353662954, 1353662948, 1353662943, 1353662937, 1353662912, 1353662909, 1353662905, 1353662902, 1353662899, 1353662894, 1353662815, 1353662811, 1353662806, 1353662801, 1353662797, 1353662791, 1353662759, 1353662757, 1353662750, 1353662744] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.04973745346069336 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 125.03509974479675 time spend to save output : 0.06492400169372559 total time spend for step 0 : 125.10002374649048 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.45user 5.61system 2:22.13elapsed 14%CPU (0avgtext+0avgdata 1938116maxresident)k 2019400inputs+160552outputs (12852major+588052minor)pagefaults 0swaps