python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 560723 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7626>200 please execute split_portfolio.py -i 23282116 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2939103'] with mtr_portfolio_ids : ['23282116'] and first list_photo_ids : [] new path : /proc/560723/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 34.28362321853638 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon May 26 17:31:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 196 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 196 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 196 wait 20 seconds inside check gpu memory 2025-05-26 17:32:11.581687: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-26 17:32:11.624398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 17:32:11.627229: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 17:32:11.666208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 17:32:11.682618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 17:32:11.687710: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 17:32:11.714422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 17:32:11.720126: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 17:32:11.772917: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 17:32:11.774705: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 17:32:11.776348: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-26 17:32:11.811186: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-26 17:32:11.813093: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7d84000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-26 17:32:11.813126: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-26 17:32:11.950440: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3bc83980 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-26 17:32:11.950535: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-26 17:32:11.952328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 17:32:11.952467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 17:32:11.952501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 17:32:11.952528: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 17:32:11.952555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 17:32:11.952582: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 17:32:11.952609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 17:32:11.952636: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 17:32:11.954260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 17:32:11.954339: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 17:32:11.955905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 17:32:11.955937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 17:32:11.955950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 17:32:11.957313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 5485 max_wait_temp : 4 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 10.747639179229736 time used to load_weights : 0.7506904602050781 0it [00:00, ?it/s] 38it [00:00, 371.22it/s] 76it [00:00, 345.38it/s]temp/1748273429_560723_1360819987_5580d9c54296a1706d10e0d1d53f85a3.jpg temp/1748273429_560723_1360819984_a5ddb037aa6a31289e9cf756d7bd1768.jpg temp/1748273429_560723_1360819960_b7899c408155d35c2fb61fdbfc2727df.jpg temp/1748273429_560723_1360819959_8551b36196817b1903675803aad955c9.jpg temp/1748273429_560723_1360819958_7f1a15408297d44fd1b9c0553aecf24b.jpg temp/1748273429_560723_1360819957_0a8b5346ee9db85e6a42665a73a20736.jpg temp/1748273429_560723_1360819956_6a7656579620428d9a60db776fe1caaa.jpg temp/1748273429_560723_1360819955_1ffc99c7017b8d04827107db1747c4f7.jpg temp/1748273429_560723_1360819899_4b71a31c043ead38895c32369bda46de.jpg temp/1748273429_560723_1360819897_12decb1c3e128c3023e1d5d0f2733e44.jpg temp/1748273429_560723_1360819895_04f4faafe2d7fb101a304e40912e4d4f.jpg temp/1748273429_560723_1360819893_9ee55998aae24634b4b1e80b9a0b0294.jpg temp/1748273429_560723_1360819891_459e27abce073b2a585ecc51129f6ad0.jpg temp/1748273429_560723_1360819888_72b2294bd589e857878d8046c7fa879f.jpg temp/1748273429_560723_1360819864_9724c46f4933747f59a76b750e88baaf.jpg temp/1748273429_560723_1360819862_80154ed51000fba5cfdd691421ddd147.jpg temp/1748273429_560723_1360819859_4d6ed7ea65155ca5186f83cd749fef07.jpg temp/1748273429_560723_1360819856_1bdab44c6877ca351dfc06737fe7877a.jpg temp/1748273429_560723_1360819853_239ae97fd48b991e545961a85600fdd4.jpg temp/1748273429_560723_1360819851_3737f9090bd55ded3cb926039eb80aa9.jpg temp/1748273429_560723_1360819831_55eb72be4bdb413311692a2cdb25d3bb.jpg temp/1748273429_560723_1360819829_3b61aef3f774b92168d7565e5eec4842.jpg temp/1748273429_560723_1360819826_1dd14b42eabcfc44e35f9731651be84a.jpg temp/1748273429_560723_1360819825_981f34734994c086f3f38c557859d44e.jpg temp/1748273429_560723_1360819821_8673e593dfb763074e70031a75b08829.jpg temp/1748273429_560723_1360819816_5499a504d5dc9cf036b28933b4c86548.jpg temp/1748273429_560723_1360819748_967e188a067552ae330eb22f9126b790.jpg temp/1748273429_560723_1360819746_0912c71f947309ebd25831a88bc75218.jpg temp/1748273429_560723_1360819726_632b20f662ded0544c3a7d8157942c8d.jpg temp/1748273429_560723_1360819698_200083b2add78523f0434482335aff3a.jpg temp/1748273429_560723_1360819691_43aaad4071758e146d26290eaa7fbb72.jpg temp/1748273429_560723_1360819689_75d641df7507eac7ae47026f800cf54a.jpg temp/1748273429_560723_1360819509_902bfc0b86e013eb88d151fcd827b109.jpg temp/1748273429_560723_1360819508_11e7c5334e0dd06873158a492a6a6f91.jpg temp/1748273429_560723_1360819505_f1a8a1daaeea9f74a549c405e23801b9.jpg temp/1748273429_560723_1360819503_50d762b4d574ac735db2a8e09ff05a05.jpg temp/1748273429_560723_1360819500_e3446c39898ae878cb3587d532948fde.jpg temp/1748273429_560723_1360819498_965dbfe8cb8d1cd78c4e48a034aa9225.jpg temp/1748273429_560723_1360819480_beadd07f70be2b13c6734ea4db028250.jpg temp/1748273429_560723_1360819478_81d15c73a3d1a34dddc33caf826eaddd.jpg temp/1748273429_560723_1360819475_beba70180a8f184f0cb3b41aa6f711a4.jpg temp/1748273429_560723_1360819432_b4483fa61bfa701d190d887cd3bdc101.jpg temp/1748273429_560723_1360819405_3552f0eed4d1a157dea05749ab125b2a.jpg temp/1748273429_560723_1360819371_3b80c9d8e32fa9d28dc8f92a5b2965fc.jpg temp/1748273429_560723_1360818564_22138d5c4d6fe36327e74e5d45a1e87d.jpg temp/1748273429_560723_1360818513_e2ec10c2c2e47da160674d058441b142.jpg temp/1748273429_560723_1360818440_21b15594a872dabf0fc3ba508520af6a.jpg temp/1748273429_560723_1360818390_3955debba650d5ac1d0d4021f197d866.jpg temp/1748273429_560723_1360818329_ddb7d86a61d9fb894211336b4e5c7e65.jpg temp/1748273429_560723_1360818264_822e49d40d8c43cb3028f9cad9b5a6f2.jpg temp/1748273429_560723_1360818127_1493b5486dba5ff89f9e616609edecef.jpg temp/1748273429_560723_1360818124_48b597e4d533ab91253ccc916aa2fd42.jpg temp/1748273429_560723_1360818120_4dcd37f881d02cfdfdcde74a2c27b713.jpg temp/1748273429_560723_1360818066_15d4db0e95b537513453370596c9e5e8.jpg temp/1748273429_560723_1360818014_8e431e0c9216d6d74565166dbb76dc39.jpg temp/1748273429_560723_1360817938_221961d916fc6bdb16e6c0d926489d93.jpg temp/1748273429_560723_1360817468_c46b51fde9cfd8b2ff2ab93a2465d6ba.jpg temp/1748273429_560723_1360817432_eae3942e822a15d3cdb8bfd721a62eca.jpg temp/1748273429_560723_1360817385_ce3eec00c12e00a0dec5c329ad226eaa.jpg temp/1748273429_560723_1360817359_5cb82459896e21e4ce81a7fc8539bfa5.jpg temp/1748273429_560723_1360817320_4ebdd3e5fd3d8c7e9e3f737a180743e7.jpg temp/1748273429_560723_1360817228_0a2ba9b73189094f295fd196c5570e89.jpg temp/1748273429_560723_1360816645_10c6936fbec4bd729d650545a00e3239.jpg temp/1748273429_560723_1360816518_9710e271543282ac86ab5460a2a157a4.jpg temp/1748273429_560723_1360816407_77a90e7d8a7a8a04626feaa8d50f2ec2.jpg temp/1748273429_560723_1360816331_dce333ef897e3136d0cdb808425f6e04.jpg temp/1748273429_560723_1360816250_c3ffd739656f9ab54a12994a1cdcd8a7.jpg temp/1748273429_560723_1360816172_388ddbf6251361b5e26991a53664dd9e.jpg temp/1748273429_560723_1360815632_0b7fef0a1af743f8b5ec9c2d5d14b33b.jpg temp/1748273429_560723_1360815594_301b44bed96756dba31e8533555509c0.jpg temp/1748273429_560723_1360815519_253d7d91fc5070d94cf313a7be147704.jpg temp/1748273429_560723_1360815491_d59e04b6cda96c74a7ecaf6d4cf232ca.jpg temp/1748273429_560723_1360815459_e9f3724cadd92e3cbfebfc709bf6bca6.jpg temp/1748273429_560723_1360815411_66b1827a0051c3f847644208baec9ed4.jpg temp/1748273429_560723_1360815069_be4dfaf4095a1d3e4f9d3e4f1586e3ea.jpg temp/1748273429_560723_1360814999_2d3dc1eaed8da9db9965c460277cde24.jpg temp/1748273429_560723_1360814942_1febf1873b0178a60459e7a317262639.jpg temp/1748273429_560723_1360814907_2bf3d5bf8366e7ae1c34851171fb92fc.jpg temp/1748273429_560723_1360814857_c6876d92b559a26bcc8befb4bfa61964.jpg temp/1748273429_560723_1360814821_f640c4a0bc7938f1b21ba65d7861ce37.jpg temp/1748273429_560723_1360814628_7b8fec73e81134b639b4fb43057928a9.jpg temp/1748273429_560723_1360814600_ba53f3c590db2f49bfe8f4821ee27f07.jpg temp/1748273429_560723_1360814551_ad66d6cf1a2547cf3d5dd31847879490.jpg temp/1748273429_560723_1360814522_165098185391fe5bcb8d80ff35029d56.jpg temp/1748273429_560723_1360814496_415c492da1822cc4460a040914659466.jpg temp/1748273429_560723_1360814425_5c0341be98f34434f10ae5186226efab.jpg temp/1748273429_560723_1360814333_7920d189ba58fc43f2c63b7e9eb5f365.jpg temp/1748273429_560723_1360814329_c8414a3cc9ded35313607b248725e78f.jpg temp/1748273429_560723_1360814316_946c853469cda178f0f1fb88f0f30e60.jpg temp/1748273429_560723_1360814301_79cbe71c2fb6a2d370873b7eb754e608.jpg temp/1748273429_560723_1360814298_f89384f42879e86ee600d9bf01725f91.jpg temp/1748273429_560723_1360814282_0441ed77089de334edc3b9804ddec67c.jpg temp/1748273429_560723_1360814198_dcd4e5c34b182e1db39848aacc6c0b52.jpg temp/1748273429_560723_1360814172_da16d2f944eeebee9719f043c6761b9e.jpg temp/1748273429_560723_1360814119_8912d33b8c3ac8c0b76fa43c4bf6921e.jpg temp/1748273429_560723_1360814090_4d55e4483d62e06c2732dbc797394ea4.jpg temp/1748273429_560723_1360814065_0e4339fce965985f4180fac01b0eb963.jpg temp/1748273429_560723_1360814062_1a30d6a1c955e6ac9ebd63757136c08b.jpg temp/1748273429_560723_1360813653_f22e5e951b707580ce631b33c4f2f6cf.jpg temp/1748273429_560723_1360813622_6a8dc78b60bf25f19a323a02c7c62e48.jpg temp/1748273429_560723_1360813594_da073bed87980e478bf6d087c870144d.jpg temp/1748273429_560723_1360813567_58479627dca242d4f691183170fd77dc.jpg temp/1748273429_560723_1360813538_d40f1e933bd5ebe46a2af4db6da16035.jpg temp/1748273429_560723_1360813489_ee3fcc0021e8277486233c86b4dde1fe.jpg temp/1748273429_560723_1360813226_0a9567c768abe98714661958c4d9feef.jpg temp/1748273429_560723_1360813199_a11c6000290d98c0825733cbff2cb07f.jpg temp/1748273429_560723_1360813189_9f76b45ef03b96adb5cb4f6cb9ab0ca2.jpg temp/1748273429_560723_1360813187_636b5df76c4cb1073339b6189c17f571.jpg temp/1748273429_560723_1360813185_802129e3aef50d8f31a8f1cc10e0085c.jpg temp/1748273429_560723_1360813184_0abfef8518ebd657c313786c118d98f9.jpg temp/1748273429_560723_1360813107_627972688995cdfa9333266f8c6e27c9.jpg temp/1748273429_560723_1360813079_65b536d814da2f32ff87760d44aec158.jpg temp/1748273429_560723_1360813052_69fd8ad216e0270bf9cb387fde442e1f.jpg temp/1748273429_560723_1360813019_d8214a51ed35f1755871bd458a9c3d95.jpg temp/1748273429_560723_1360812992_710ca2802d36202fcd8c312419b6611f.jpg 119it [00:00, 375.96it/s] 157it [00:00, 281.99it/s] 200it [00:00, 340.04it/s]2025-05-26 17:32:26.784248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 17:32:26.812877: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-26 17:32:26.878168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 17:32:26.906894: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-26 17:32:26.914627: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748273429_560723_1360812965_010ff36c58d207b264da56818286ba2f.jpg temp/1748273429_560723_1360812705_351fd1d8d92bb3d092dada9a0dfda4d6.jpg temp/1748273429_560723_1360812700_605ee335f5f121a6b7d837cb82c8a48e.jpg temp/1748273429_560723_1360812683_c680533f0a6edef2c1f66f52e6e70b8b.jpg temp/1748273429_560723_1360812648_c7270ff481820a4a6c9e0fcbfd58bdbd.jpg temp/1748273429_560723_1360812575_73a39185c2875f52a10a89ee9789488d.jpg temp/1748273429_560723_1360812525_6d6ad8d8c9e7ea4baec02001fa17be33.jpg temp/1748273429_560723_1360812353_3497a9e2fcd4d515400390bff78c5d39.jpg temp/1748273429_560723_1360812319_8bc92ba587bf1d8711feabb6266999e0.jpg temp/1748273429_560723_1360812262_0bee19c9bac31f19bfcb91339bf1e06c.jpg temp/1748273429_560723_1360812204_e32a930a1c5a658af4a9623bc99e4f7f.jpg temp/1748273429_560723_1360812165_d6fbdf46e6021a633f6da9dfd0325435.jpg temp/1748273429_560723_1360812139_71e80a25113adc7077c92328e0b7a6e7.jpg temp/1748273429_560723_1360811871_747c1cb1b026f06bafb3d070c1174c09.jpg temp/1748273429_560723_1360811837_73dae6b467d752e4f5ec583532d9e3ac.jpg temp/1748273429_560723_1360811807_0d0a3d603b11e3771e50ddae6c52a838.jpg temp/1748273429_560723_1360811779_3c348606d119c5cdce1bd14576c841de.jpg temp/1748273429_560723_1360811772_c2e8373be4476ada987980c209767eed.jpg temp/1748273429_560723_1360811758_c8cd3ee894f123478df913a1fcfba1ce.jpg temp/1748273429_560723_1360811495_93df5c45f4110b3a65862d9faf52192d.jpg temp/1748273429_560723_1360811465_fbb9804ddcd5d41540e1e6611089a93b.jpg temp/1748273429_560723_1360811438_6a16df1319ddac1cd0c7a9877041cb10.jpg temp/1748273429_560723_1360811413_fa6d7be960b06b504d5181afdde7f9ea.jpg temp/1748273429_560723_1360811386_5410ce0435a23db8ae4749a5c3a71936.jpg temp/1748273429_560723_1360811354_09a68b0ed5f0840023831f5e2415d3d4.jpg temp/1748273429_560723_1360811134_f92645af282f6274b973a7c12dbe2c6a.jpg temp/1748273429_560723_1360811131_9654375b7fdc570271f4184f957f7104.jpg temp/1748273429_560723_1360811128_ca51b3f4328abe751e24fb8d908b4128.jpg temp/1748273429_560723_1360811109_ddf82dd2bab9ce315185733d4cadd934.jpg temp/1748273429_560723_1360811095_23c294f3abd644902dcaa6a2defbca8c.jpg temp/1748273429_560723_1360811080_335606268e04cf288caf3a922dfaa5ed.jpg temp/1748273429_560723_1360810939_45b2a770d040b15b6acce115b6b77268.jpg temp/1748273429_560723_1360810911_f46d7579190febd678d9071ed7b885a0.jpg temp/1748273429_560723_1360810905_658d0adb3585a5381608c08862c193c9.jpg temp/1748273429_560723_1360810893_c655083f43d0b06cf92cf06c33ef6b8e.jpg temp/1748273429_560723_1360810878_52c5b3aa49eb2640ddc223498ba8f6af.jpg temp/1748273429_560723_1360810851_26eab457a111352a5b49f17c402e84ab.jpg temp/1748273429_560723_1360809662_d33118005df78563de051eda30a90cb6.jpg temp/1748273429_560723_1360809633_bff19ced982bb2b72f2cbdefe73d59fb.jpg temp/1748273429_560723_1360809606_3ed842918cd8c8e5b86899149e481473.jpg temp/1748273429_560723_1360809559_e109208b50ff031ed6e50cbde371c247.jpg temp/1748273429_560723_1360809505_c83e1aa7e77d2afa65f2a0deb600fbac.jpg temp/1748273429_560723_1360809445_babc4a5b76652647a610cfeffb28b993.jpg temp/1748273429_560723_1360809102_c1691ee9f5cb18151ccb907d4f4096fa.jpg temp/1748273429_560723_1360809069_936e010feafa0888c797328cb4ef84e6.jpg temp/1748273429_560723_1360809040_16cbe17295baab2b7e210464bae42312.jpg temp/1748273429_560723_1360809010_97a13fd99389fd743c03dc485f254b52.jpg temp/1748273429_560723_1360808973_058cdefa3f241c502069b429def449dd.jpg temp/1748273429_560723_1360808901_b23cfda1fb5772089c99740b40f45d1a.jpg temp/1748273429_560723_1360808543_1e245ba82bd41b12e342656e667e0118.jpg temp/1748273429_560723_1360808416_37a1d8f9801b30a1cc36fa5e5725f076.jpg temp/1748273429_560723_1360808358_b66492e3048c05b88c0c40aeb0327234.jpg temp/1748273429_560723_1360808257_ba4b07688d595e87c802dd8e0d07da46.jpg temp/1748273429_560723_1360808224_be965ca71b5c2c08218f0db5ccccd71d.jpg temp/1748273429_560723_1360808197_d7b44b6a60202842b36f596537756ac0.jpg temp/1748273429_560723_1360807972_aec5c93a46ec2d0e365f4a46c465d79c.jpg temp/1748273429_560723_1360807887_f6e7ad73e54642fa17067c87d79ff3a9.jpg temp/1748273429_560723_1360807834_2e610a6d3cf28d258ebbe741aa62f449.jpg temp/1748273429_560723_1360807781_d58f728faf51df7ba2cae632d0bf807e.jpg temp/1748273429_560723_1360807729_e1d5291451e86ba96ac365c6cc984336.jpg temp/1748273429_560723_1360807684_f0193d26f8bbc6736892532b0aae01a0.jpg temp/1748273429_560723_1360807382_151734ab22461845e100a6de979e9bd4.jpg temp/1748273429_560723_1360807347_7735bf717569d2aeee6c15f3e8641cbd.jpg temp/1748273429_560723_1360807299_e2fc541c34db1cb76a4d1f112eab81a8.jpg temp/1748273429_560723_1360807270_79850e6b938fd17653b460aabd9c74d5.jpg temp/1748273429_560723_1360807228_b7787e5e569c1d0f0d09d89e13f26e70.jpg temp/1748273429_560723_1360807160_6388f16f75b5f63c9d5c1a781e0cfa29.jpg temp/1748273429_560723_1360806567_2a29c01d797fa4646994c6c5c70a1ed1.jpg temp/1748273429_560723_1360806490_c4b3a7a83b534f8c0456ccb97164a56e.jpg temp/1748273429_560723_1360806414_c3ff2ae212a55bf6e3adac5a43a0b9a6.jpg temp/1748273429_560723_1360806353_db8cc57479b6e0c24ca2888785eb627d.jpg temp/1748273429_560723_1360806297_83919df1c0112d5a6ad12f9a5a7cfcd6.jpg temp/1748273429_560723_1360806241_02269eae277cc2a54962247993032aed.jpg temp/1748273429_560723_1360805948_843e30b23d359e769d14611821ca1372.jpg temp/1748273429_560723_1360805920_95a290356fc04f2458f7089924c4395f.jpg temp/1748273429_560723_1360805859_02a2f274680ce4c17575b58fab3b4388.jpg temp/1748273429_560723_1360805802_5d192ca7e5019c133ded67086b9d7328.jpg temp/1748273429_560723_1360805774_b935a15c405b2f5c0112714e75e90e2c.jpg temp/1748273429_560723_1360805771_b8bd0a4d1fabefc7e36cd5472694fcc2.jpg temp/1748273429_560723_1360805577_23f6666e1fd6626104475101c827c49e.jpg temp/1748273429_560723_1360805526_b75ec2c08edb2e2380cd1d8d519a4a3d.jpg temp/1748273429_560723_1360805450_c03bb6212e12b49234024193e7aa156d.jpg temp/1748273429_560723_1360805397_8c23f2a0577b81e4cf715e28385de925.jpg temp/1748273429_560723_1360805341_9033ad91bd27f53f412b35cd88125988.jpg temp/1748273429_560723_1360805291_98a664a4f48165332f2a35f646120d25.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1360819987, 1360819984, 1360819960, 1360819959, 1360819958, 1360819957, 1360819956, 1360819955, 1360819899, 1360819897, 1360819895, 1360819893, 1360819891, 1360819888, 1360819864, 1360819862, 1360819859, 1360819856, 1360819853, 1360819851, 1360819831, 1360819829, 1360819826, 1360819825, 1360819821, 1360819816, 1360819748, 1360819746, 1360819726, 1360819698, 1360819691, 1360819689, 1360819509, 1360819508, 1360819505, 1360819503, 1360819500, 1360819498, 1360819480, 1360819478, 1360819475, 1360819432, 1360819405, 1360819371, 1360818564, 1360818513, 1360818440, 1360818390, 1360818329, 1360818264, 1360818127, 1360818124, 1360818120, 1360818066, 1360818014, 1360817938, 1360817468, 1360817432, 1360817385, 1360817359, 1360817320, 1360817228, 1360816645, 1360816518, 1360816407, 1360816331, 1360816250, 1360816172, 1360815632, 1360815594, 1360815519, 1360815491, 1360815459, 1360815411, 1360815069, 1360814999, 1360814942, 1360814907, 1360814857, 1360814821, 1360814628, 1360814600, 1360814551, 1360814522, 1360814496, 1360814425, 1360814333, 1360814329, 1360814316, 1360814301, 1360814298, 1360814282, 1360814198, 1360814172, 1360814119, 1360814090, 1360814065, 1360814062, 1360813653, 1360813622, 1360813594, 1360813567, 1360813538, 1360813489, 1360813226, 1360813199, 1360813189, 1360813187, 1360813185, 1360813184, 1360813107, 1360813079, 1360813052, 1360813019, 1360812992, 1360812965, 1360812705, 1360812700, 1360812683, 1360812648, 1360812575, 1360812525, 1360812353, 1360812319, 1360812262, 1360812204, 1360812165, 1360812139, 1360811871, 1360811837, 1360811807, 1360811779, 1360811772, 1360811758, 1360811495, 1360811465, 1360811438, 1360811413, 1360811386, 1360811354, 1360811134, 1360811131, 1360811128, 1360811109, 1360811095, 1360811080, 1360810939, 1360810911, 1360810905, 1360810893, 1360810878, 1360810851, 1360809662, 1360809633, 1360809606, 1360809559, 1360809505, 1360809445, 1360809102, 1360809069, 1360809040, 1360809010, 1360808973, 1360808901, 1360808543, 1360808416, 1360808358, 1360808257, 1360808224, 1360808197, 1360807972, 1360807887, 1360807834, 1360807781, 1360807729, 1360807684, 1360807382, 1360807347, 1360807299, 1360807270, 1360807228, 1360807160, 1360806567, 1360806490, 1360806414, 1360806353, 1360806297, 1360806241, 1360805948, 1360805920, 1360805859, 1360805802, 1360805774, 1360805771, 1360805577, 1360805526, 1360805450, 1360805397, 1360805341, 1360805291] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.09415888786315918 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 83.07346868515015 time spend to save output : 0.10482501983642578 total time spend for step 0 : 83.17829370498657 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 17.26user 5.55system 2:03.01elapsed 18%CPU (0avgtext+0avgdata 1749808maxresident)k 1406584inputs+270304outputs (7942major+517748minor)pagefaults 0swaps