python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2855415 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 196>100 please execute split_portfolio.py -i 23049130 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2873065'] with mtr_portfolio_ids : ['23049130'] and first list_photo_ids : [] new path : /proc/2855415/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 11.989240646362305 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri May 16 08:40:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-16 08:40:43.817146: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-16 08:40:43.846298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 08:40:43.847127: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 08:40:43.850275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 08:40:43.853210: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 08:40:43.854062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 08:40:43.857260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 08:40:43.859293: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 08:40:43.865631: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 08:40:43.867775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 08:40:43.868294: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-16 08:40:43.887463: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-16 08:40:43.890225: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f5ee4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-16 08:40:43.890280: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-16 08:40:44.039519: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x124d0770 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-16 08:40:44.039841: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-16 08:40:44.041020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 08:40:44.041163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 08:40:44.041192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 08:40:44.041216: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 08:40:44.041240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 08:40:44.041262: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 08:40:44.041284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 08:40:44.041307: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 08:40:44.042860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 08:40:44.042999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 08:40:44.044020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-16 08:40:44.044041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-16 08:40:44.044052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-16 08:40:44.045635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 9352 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 10.285162210464478 time used to load_weights : 0.7825767993927002 0it [00:00, ?it/s] 32it [00:00, 284.85it/s] 77it [00:00, 376.82it/s] 100it [00:00, 414.75it/s]2025-05-16 08:40:58.483474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 08:40:58.521841: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-16 08:40:58.596408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 08:40:58.604194: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-16 08:40:58.607917: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1747377628_2855415_1358860544_853f6618debafc3c14b43e459f5ef00e.jpg temp/1747377628_2855415_1358860542_10e3b88ad66041b21b45ef56cbfdea51.jpg temp/1747377628_2855415_1358860498_96c30be48a55a863db77cc1ad55104bb.jpg temp/1747377628_2855415_1358860496_5104b944176884d308c1a3e6978dc24e.jpg temp/1747377628_2855415_1358860446_a980644b7a229fdb2253623202efcf9a.jpg temp/1747377628_2855415_1358860442_6c3e463c8a3fa3bdb42864c8225000d7.jpg temp/1747377628_2855415_1358860396_252194453944c18a978b3777ff85e02f.jpg temp/1747377628_2855415_1358860393_43e9b47bfdd63b864c97239de8385a9b.jpg temp/1747377628_2855415_1358860342_ec15512c3eb50a8c24b6f07d1cbb2dce.jpg temp/1747377628_2855415_1358860340_95f5b10d7e343448375681cb8faca06e.jpg temp/1747377628_2855415_1358860288_7e3743fb261ece113fc1d2ef9cdda28c.jpg temp/1747377628_2855415_1358860284_86d03ae6cad7c4efd7dbfe798650efd4.jpg temp/1747377628_2855415_1358860257_66308cb40bc0f4ef2fc1926ce489ea63.jpg temp/1747377628_2855415_1358860255_9fc6fba73992328530bd5a198efe4f16.jpg temp/1747377628_2855415_1358860198_7facce172c51c6be446df135aafb3906.jpg temp/1747377628_2855415_1358860193_9cfa68d6db763a71cefd89cea4cadfd2.jpg temp/1747377628_2855415_1358860153_ec2fbf9720fa566cab13b14f62a5017a.jpg temp/1747377628_2855415_1358860149_a982824233c143d2752e8e11d261fbbb.jpg temp/1747377628_2855415_1358860059_983e44c5601b9e01948861d0be22e2fa.jpg temp/1747377628_2855415_1358860056_441e478b8996ad683a15a91cb9064bbb.jpg temp/1747377628_2855415_1358859931_ab357f04fcbc7072eaef486675e1fd6f.jpg temp/1747377628_2855415_1358859910_ddc0d55e172b3d87e568696fc804f7ad.jpg temp/1747377628_2855415_1358859838_9f962f67eefe1420a709abb6ee72c84e.jpg temp/1747377628_2855415_1358859836_82a6a01d136f2871f7a7f4098946416d.jpg temp/1747377628_2855415_1358859754_09b062bdd6bf6252309f6b1d77f453b1.jpg temp/1747377628_2855415_1358859747_176e3e6464b0d085fb0c122f1e5ea9e6.jpg temp/1747377628_2855415_1358859494_50a6bbb8844aca0e9bc2eb28d9f23f0d.jpg temp/1747377628_2855415_1358859467_385452c46b26886312c4d014cb21247f.jpg temp/1747377628_2855415_1358859255_f4d1adb2d4ed0f1fe8b6e0c57c9a5f66.jpg temp/1747377628_2855415_1358859253_54e26428fa940c1b12e78dbc55e57990.jpg temp/1747377628_2855415_1358859204_67b9ec9b010fc4b8a4c41ea5ce0ea1b0.jpg temp/1747377628_2855415_1358859180_c591d6c7ec62294c21170940ff24961b.jpg temp/1747377628_2855415_1358859135_91b991bf15e00c6e2a0fac65846d1cff.jpg temp/1747377628_2855415_1358859133_d80bb2237751445836e95e1dfd929add.jpg temp/1747377628_2855415_1358859084_0bfcc78dc557431bb72d074501c4c304.jpg temp/1747377628_2855415_1358859080_4b212f90123a9d9139a085d28a34b1c9.jpg temp/1747377628_2855415_1358859024_7aa8486fe04e1ab2613d937b09b4d392.jpg temp/1747377628_2855415_1358859020_04f2f695d42b9dec4c260fa09e949435.jpg temp/1747377628_2855415_1358858982_f31f15ff25aca2a9b54222999e53c570.jpg temp/1747377628_2855415_1358858978_f8db9e508ff580088b12f3b5b3808ca8.jpg temp/1747377628_2855415_1358858943_1c5424f051a29dd99de1f81e92a4ff76.jpg temp/1747377628_2855415_1358858938_20ea823ac79e3f153617664c8943ec80.jpg temp/1747377628_2855415_1358858902_c2949ef52910d988a279c88450201a27.jpg temp/1747377628_2855415_1358858899_f5a60d9e5bf2e33dc7ae8b0d19ee39c2.jpg temp/1747377628_2855415_1358858624_ab3a34e0c36854e0fd4d4103f53373a9.jpg temp/1747377628_2855415_1358858619_17a62485ec1bc10b56ab74e3c087e97a.jpg temp/1747377628_2855415_1358858568_d09db67d2d73f258590dc89166bd3881.jpg temp/1747377628_2855415_1358858567_97666efc05ae4fae37c8e0681c348e43.jpg temp/1747377628_2855415_1358858465_b78e69921519da7e9e05da494bd142fb.jpg temp/1747377628_2855415_1358858458_c38f15e39ea7f0d0e12743b757ba886a.jpg temp/1747377628_2855415_1358858233_4050c6d676685e5bddf64b48e0c7c76e.jpg temp/1747377628_2855415_1358858229_f6f777242b31c63dced9376423638904.jpg temp/1747377628_2855415_1358858001_2f7ebd671f123eb523f42d7a7fcbf5aa.jpg temp/1747377628_2855415_1358857957_104ba739c78974018628648b2c610bf1.jpg temp/1747377628_2855415_1358857547_10d988955c786c67de027c563dc5c2e0.jpg temp/1747377628_2855415_1358857446_daf80f3fb064fb8882c51dbf3f94d088.jpg temp/1747377628_2855415_1358856809_c01f3d000a7e5e3c437c9e327e61fb50.jpg temp/1747377628_2855415_1358856806_11fd2ab8c6b374d8037730c5f85885cc.jpg temp/1747377628_2855415_1358856514_e6e488ea279c65670eb13bfaf48a6813.jpg temp/1747377628_2855415_1358856482_c558d58b47fba4b1f5e161ab64118436.jpg temp/1747377628_2855415_1358855997_f03a8dc6e94be9dcba006e2543968778.jpg temp/1747377628_2855415_1358855939_1ea968ffbcdae1110881be5423707bbe.jpg temp/1747377628_2855415_1358855079_a1bf389608352b38920c808b7218bfd0.jpg temp/1747377628_2855415_1358854819_504f85953368652d326a33d105be1ae6.jpg temp/1747377628_2855415_1358854071_cc43fe5a21d3c78ba19fdcf038e12011.jpg temp/1747377628_2855415_1358854005_a4b5906a141933ae0f74312cc3c71c0c.jpg temp/1747377628_2855415_1358853558_511ccda712979f90243bd1445411adba.jpg temp/1747377628_2855415_1358853521_f92a22f3d68b19c50d90798e347b56f0.jpg temp/1747377628_2855415_1358853106_1bf0a3a29460e1258c68c191e2508f74.jpg temp/1747377628_2855415_1358853077_843c3c6e153474ce2c0917d64404bbdd.jpg temp/1747377628_2855415_1358852865_bcc4618167d63b60297eb235d69a3443.jpg temp/1747377628_2855415_1358852858_4a336e0f29ec1a0000d68b021c5a99aa.jpg temp/1747377628_2855415_1358852599_083574e1c457870522d30b61557fff3e.jpg temp/1747377628_2855415_1358852594_0e193545826f1be17d1abd6587aad0ae.jpg temp/1747377628_2855415_1358852324_a14525c105f1949869c4e5007390ed5c.jpg temp/1747377628_2855415_1358852296_05b33468f93903ca9edf58e534817b01.jpg temp/1747377628_2855415_1358851917_e415d0e4f3d8879304e5d30b6ac09ebb.jpg temp/1747377628_2855415_1358851888_37d4cc9c53a55ba2417caa64aa17b80b.jpg temp/1747377628_2855415_1358851743_b5933d1313fb8bd0849d477842241e78.jpg temp/1747377628_2855415_1358851739_b0a2c72d0ecfc5d2c75ecb579d039ab3.jpg temp/1747377628_2855415_1358851709_ac387260cf11268c10ed128679e897fe.jpg temp/1747377628_2855415_1358851700_6b719185daad922d53caa05dc9c0f772.jpg temp/1747377628_2855415_1358851666_b0edd5878024a1a2e4d907089f007940.jpg temp/1747377628_2855415_1358851665_0de0cf9d3a7a150e95375f00a81c8699.jpg temp/1747377628_2855415_1358851624_abbb5b3c59a36fa99a714af619b5f69f.jpg temp/1747377628_2855415_1358851618_cb90302c881dda28ba677087f11bfee2.jpg temp/1747377628_2855415_1358851578_c4bb6b8e06d994ed7195e9e5473afaca.jpg temp/1747377628_2855415_1358851575_55f9f2d8cbcd8d51a68df0fe988f16e7.jpg temp/1747377628_2855415_1358851428_dbdfcf20e68bc02f64351db4596ba4af.jpg temp/1747377628_2855415_1358851406_6984dc1eca12a76e25ebb61806fc05b9.jpg temp/1747377628_2855415_1358851194_161483a22c951c9f9dc681cf82a93524.jpg temp/1747377628_2855415_1358851143_316d7157723f7b8f75bf867ead249276.jpg temp/1747377628_2855415_1358850767_be70c1b0c486b6bcc859b11c5ab6a00a.jpg temp/1747377628_2855415_1358850691_d65362a0d314b8e7bb744999d51b74aa.jpg temp/1747377628_2855415_1358850205_0caa4a18d38b473aea1c2a1164346b69.jpg temp/1747377628_2855415_1358850099_a6e942fe9104afb6446ec88e4d4ab0d9.jpg temp/1747377628_2855415_1358849731_7c3cd506d838ce42344672798a83650c.jpg temp/1747377628_2855415_1358849703_14bd3afc2f3617a18d83d31f0fb7a787.jpg temp/1747377628_2855415_1358849345_8027ec76432696a03165b3b002bf39fe.jpg temp/1747377628_2855415_1358848632_4fead9f0f46217b1bcc9cef5f8e73d0a.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1358860544, 1358860542, 1358860498, 1358860496, 1358860446, 1358860442, 1358860396, 1358860393, 1358860342, 1358860340, 1358860288, 1358860284, 1358860257, 1358860255, 1358860198, 1358860193, 1358860153, 1358860149, 1358860059, 1358860056, 1358859931, 1358859910, 1358859838, 1358859836, 1358859754, 1358859747, 1358859494, 1358859467, 1358859255, 1358859253, 1358859204, 1358859180, 1358859135, 1358859133, 1358859084, 1358859080, 1358859024, 1358859020, 1358858982, 1358858978, 1358858943, 1358858938, 1358858902, 1358858899, 1358858624, 1358858619, 1358858568, 1358858567, 1358858465, 1358858458, 1358858233, 1358858229, 1358858001, 1358857957, 1358857547, 1358857446, 1358856809, 1358856806, 1358856514, 1358856482, 1358855997, 1358855939, 1358855079, 1358854819, 1358854071, 1358854005, 1358853558, 1358853521, 1358853106, 1358853077, 1358852865, 1358852858, 1358852599, 1358852594, 1358852324, 1358852296, 1358851917, 1358851888, 1358851743, 1358851739, 1358851709, 1358851700, 1358851666, 1358851665, 1358851624, 1358851618, 1358851578, 1358851575, 1358851428, 1358851406, 1358851194, 1358851143, 1358850767, 1358850691, 1358850205, 1358850099, 1358849731, 1358849703, 1358849345, 1358848632] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.1583867073059082 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 18.03822088241577 time spend to save output : 0.16841435432434082 total time spend for step 0 : 18.206635236740112 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 16.20user 4.53system 0:36.22elapsed 57%CPU (0avgtext+0avgdata 1740880maxresident)k 463944inputs+92296outputs (130major+693228minor)pagefaults 0swaps