python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 327365 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2369>200 please execute split_portfolio.py -i 27957750 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3966799'] with mtr_portfolio_ids : ['27957750'] and first list_photo_ids : [] new path : /proc/327365/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 36.36457133293152 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri Oct 31 06:51:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 73 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 73 wait 20 seconds inside check gpu memory 2025-10-31 06:51:49.865364: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-10-31 06:51:49.894759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-31 06:51:49.895013: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-31 06:51:49.896844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-31 06:51:49.898763: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-31 06:51:49.899086: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-31 06:51:49.901209: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-31 06:51:49.902418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-31 06:51:49.906672: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-31 06:51:49.907791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-31 06:51:49.908129: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-10-31 06:51:49.934550: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-10-31 06:51:49.936459: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8e70000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-10-31 06:51:49.936498: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-10-31 06:51:50.059365: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x60a40a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-10-31 06:51:50.059424: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-10-31 06:51:50.060012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-31 06:51:50.060070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-31 06:51:50.060088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-31 06:51:50.060104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-31 06:51:50.060119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-31 06:51:50.060135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-31 06:51:50.060150: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-31 06:51:50.060166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-31 06:51:50.061036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-31 06:51:50.061077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-31 06:51:50.061618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-10-31 06:51:50.061633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-10-31 06:51:50.061642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-10-31 06:51:50.062592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3793 max_wait_temp : 3 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.56662654876709 time used to load_weights : 0.26493263244628906 0it [00:00, ?it/s] 60it [00:00, 595.55it/s]temp/1761889828_327365_1390971625_6fef58eaf158e6eeaddc2ac4d3b123db.jpg temp/1761889828_327365_1390971418_969889f1832efc59c46bde859dd4bebc.jpg temp/1761889828_327365_1390971202_ce629c35442a60b0f864d62e97f0679f.jpg temp/1761889828_327365_1390970993_035a8a0f45bc76d4993309c1e577e327.jpg temp/1761889828_327365_1390970915_c0ebba39daf3a343b06b878289af1422.jpg temp/1761889828_327365_1390970869_b47591748cb7b6022984fb91356ab81f.jpg temp/1761889828_327365_1390970532_1ef83d979e3570933f0549a0cccc2492.jpg temp/1761889828_327365_1390970523_d3744fd9278f24a01c9ca0cafab57586.jpg temp/1761889828_327365_1390970505_e42b29e99aa07aa02383e63d1504ac14.jpg temp/1761889828_327365_1390970501_8c751ec246f91ab5e82444ed6f6d0e15.jpg temp/1761889828_327365_1390970498_3d41c6b67a5fa930df6b0c66973a0f69.jpg temp/1761889828_327365_1390970495_e22cb579114c13a9fd88937e15b441bf.jpg temp/1761889828_327365_1390970236_a771118613d4d4171c6ade5970061bf0.jpg temp/1761889828_327365_1390970213_5487a5f769252e89c9c885cd10a421ea.jpg temp/1761889828_327365_1390970210_6f3906aeb5426fbf6d966aef36c08232.jpg temp/1761889828_327365_1390970208_b9b059dac1924c271bf7ca796d6780d7.jpg temp/1761889828_327365_1390969630_784eb0d2675134f459477002591755dc.jpg temp/1761889828_327365_1390969589_a49fc1718b213d415fc6a40bf12968cb.jpg temp/1761889828_327365_1390969546_08629dd087d98edf3d2440892df65666.jpg temp/1761889828_327365_1390969504_3f887344383c8cd2a39bed9ee259797a.jpg temp/1761889828_327365_1390969463_b08e353107b0be845563011e944f61c0.jpg temp/1761889828_327365_1390969420_3873c29463ef25e77bd2c4e05f2fee8d.jpg temp/1761889828_327365_1390968831_900ca90c0eb6d0961c126ad5a9eee243.jpg temp/1761889828_327365_1390968788_3ddddc9dbfda41e517050299eea05eb7.jpg temp/1761889828_327365_1390968735_58cbc4f24eb72bdb915773c9ce8132cd.jpg temp/1761889828_327365_1390968723_9f9b8f913c88a26eea53e51531575490.jpg temp/1761889828_327365_1390968718_cd5e56da0b59cb8a4a25f764f6d85b74.jpg temp/1761889828_327365_1390968714_c686a7c57302b57812d50a1c13d2b4a7.jpg temp/1761889828_327365_1390966694_559e5e2024366f86ef6d99376f2643bc.jpg temp/1761889828_327365_1390966654_b67696a616b6330008c8bc53ed70d57e.jpg temp/1761889828_327365_1390966579_55ae55325014b0a0c4338c2a1863525f.jpg temp/1761889828_327365_1390966538_f0886b3955f93685cba08cf627e064f1.jpg temp/1761889828_327365_1390966447_409e90704196949f070ed7e8deb1655c.jpg temp/1761889828_327365_1390966400_e996197c0c615d8e268c5c72618299d0.jpg temp/1761889828_327365_1390964239_0eaf9e0666503044ce3c5a701efd11d8.jpg temp/1761889828_327365_1390964236_ca200c2e3fcff9f8753a33b9608a3150.jpg temp/1761889828_327365_1390964231_9e742f27c933ed59d53e5e3e7739cdb0.jpg temp/1761889828_327365_1390964217_7d52646f317becae2c5be7fa6f053f79.jpg temp/1761889828_327365_1390964176_beade2744af9235cf5ee0621ac977c1c.jpg temp/1761889828_327365_1390964105_42f771d17ce4ebe691105782d2e9d605.jpg temp/1761889828_327365_1390958379_6cf432c1367ce0a2ad9350c289939389.jpg temp/1761889828_327365_1390958353_31dfefa51acecac65f975d31ab5f4863.jpg temp/1761889828_327365_1390958289_22cc8c0f05945e990fb1e83cdc37d3be.jpg temp/1761889828_327365_1390958231_4c4fe7a51db2714dfc203fbdee70d396.jpg temp/1761889828_327365_1390958187_1becb7fbc10e5466c43adb05855bcd1d.jpg temp/1761889828_327365_1390958146_ff601f83f14511b003c0cb322015a6a9.jpg temp/1761889828_327365_1390956694_7a31214dbb2a06d6d17589bd83f0b80b.jpg temp/1761889828_327365_1390956677_82edada45aeb3deedb7afacd668ca7f2.jpg temp/1761889828_327365_1390956501_029e135c750a06927d5dcece4aeb45d1.jpg temp/1761889828_327365_1390956458_1a50d69063c5ae652597be5618b7fb2a.jpg temp/1761889828_327365_1390956424_96d5902c2cce6c407fdd6c5fc8c328fb.jpg temp/1761889828_327365_1390956413_663881b56a1118750aa45c2c17e79c42.jpg temp/1761889828_327365_1390956409_0a657f8ca6515944f356b6ebed0129e4.jpg temp/1761889828_327365_1390956374_bccf998cf8fafa4bbcfe591f76092c75.jpg temp/1761889828_327365_1390956328_7a74b08de220ba93f5d105b087d71922.jpg temp/1761889828_327365_1390956285_fcbe30c88c99aa307125d3cdc2521143.jpg temp/1761889828_327365_1390956245_c04a2307225afeeee2b1eef1d9275e6f.jpg temp/1761889828_327365_1390956202_2a7f94adaf26a7c644ff8f27043cbed7.jpg temp/1761889828_327365_1390956163_1077ef0ccd54eb0fcb4dc2d34e4e33a9.jpg temp/1761889828_327365_1390955285_f973a1240e1c5ce2f3e3d85439d6eff3.jpg temp/1761889828_327365_1390955280_5422413f997a564f1bf6febb5f156785.jpg temp/1761889828_327365_1390955269_6e5e6bbe175649e00b114abe4e290827.jpg temp/1761889828_327365_1390955261_1693dac2abc6b049659b98bc8ffd3687.jpg temp/1761889828_327365_1390955257_e7d0dc6d33cdb3c8a3b9f494e5212037.jpg temp/1761889828_327365_1390955253_ac7d980ddcf91c1cb55359cb458beb27.jpg temp/1761889828_327365_1390954269_66a0dba492b5c9f07ef96ed13fd1d8aa.jpg temp/1761889828_327365_1390953498_9fa908975b5ac2db99a9985e41cb6aad.jpg temp/1761889828_327365_1390953454_230ba27a6c1ddb9aa1e93ad4e62ebe25.jpg temp/1761889828_327365_1390953408_4cfcf25c6020246b9eb1886b6e06b9c6.jpg temp/1761889828_327365_1390953380_3b6f4143cace9bee6f8a7a980236b738.jpg temp/1761889828_327365_1390953378_0b3fb225b40d3c54983c6fa885174824.jpg temp/1761889828_327365_1390953374_a210452b9ade651bb93c9fdf0f4a9eb4.jpg temp/1761889828_327365_1390952946_8369dfffb6670a1de18e93e221da7369.jpg temp/1761889828_327365_1390952929_d57868a946b16e5ed60f37eebabb3a36.jpg temp/1761889828_327365_1390952868_524114a1fd9eaf20688947aee6ec61c5.jpg temp/1761889828_327365_1390952843_28022ab830d79ae7b03ed0e5407e2e0b.jpg temp/1761889828_327365_1390952840_074a60cf95d84057c6b805a6b6321e19.jpg temp/1761889828_327365_1390952818_88c02e14b819bd75eb18058e3f5e306a.jpg temp/1761889828_327365_1390952748_7bdc73d1e25b8f37fecb9f14cc431d4d.jpg temp/1761889828_327365_1390952699_68c983e0bf3a3740fbd955d9eda6ed08.jpg temp/1761889828_327365_1390950658_ab59cc924ec96180a4776892d6dba9ba.jpg temp/1761889828_327365_1390950656_2df73c433d2d06ffc02da0e623c84226.jpg temp/1761889828_327365_1390950646_00354098b8acf90967d5788e5eb550c0.jpg temp/1761889828_327365_1390950634_a627714a78ff83faa751b8fe46d16527.jpg temp/1761889828_327365_1390950596_572135ae52d0cc572c98f9794796867e.jpg temp/1761889828_327365_1390950556_beb226daae8c7b8ee5f2637b20d4a147.jpg temp/1761889828_327365_1390949204_869165bb96339b68821f088f1ab6d2ba.jpg temp/1761889828_327365_1390949198_0e415fad9dba122d120a945b29df236e.jpg temp/1761889828_327365_1390949195_2c5f78ba11a5339bcd54c36b9fbe5594.jpg temp/1761889828_327365_1390949192_d799592c7f5a2deefb5ef5cee62a5b48.jpg temp/1761889828_327365_1390949188_75651e1e76d9cbca0991d3eac2682b5a.jpg temp/1761889828_327365_1390949175_197b209efedb61c380da514f28ac3732.jpg temp/1761889828_327365_1390948461_3bccbbdff89bb5dbf93ff890085e1dc2.jpg temp/1761889828_327365_1390948415_23cfe544a7d1749b8743c354e8b588dd.jpg temp/1761889828_327365_1390948364_728932cf3145b8f2d422e31ebde8c1a2.jpg temp/1761889828_327365_1390948320_5b89ba014511eefafeb696c1108ea791.jpg temp/1761889828_327365_1390948273_d1a1af1f84673b83925f013ef47898ca.jpg temp/1761889828_327365_1390948227_e813e07eb8ff8111338b1d9931304e1f.jpg temp/1761889828_327365_1390947100_aea379b2e23f32b3536f8f95a0d38164.jpg temp/1761889828_327365_1390947032_347bac94e92baf03c0c1adcff8c753f5.jpg temp/1761889828_327365_1390944762_7eee0971b90c83a24a47d8115199bce1.jpg temp/1761889828_327365_1390944716_f149d101ac0c54ba8f3ddc76021c9528.jpg temp/1761889828_327365_1390944669_74be1e60fdbf2ae4ddfc985244b055ce.jpg temp/1761889828_327365_1390944613_8bb10ff5b37f119acfdc84d8c0207647.jpg temp/1761889828_327365_1390944571_5ee5a23eaab2613f888b1c3b20b69f2b.jpg temp/1761889828_327365_1390944565_9839ac1858db4359f340f5734d6ec541.jpg temp/1761889828_327365_1390944518_60f3dfab973a33c513073919638d9be4.jpg temp/1761889828_327365_1390944487_040529ebe4f4a38398fb46e314e284e2.jpg temp/1761889828_327365_1390944483_70ad28e3658fe54e363cb74c8ef27fc6.jpg temp/1761889828_327365_1390944478_8b3163c98f44e0a01af5f1f8b1ab05c1.jpg temp/1761889828_327365_1390944475_6126ce497b8c0df0d1195d244389a5d4.jpg temp/1761889828_327365_1390944472_0f6d188d7318025f6806bc78887736f4.jpg temp/1761889828_327365_1390944443_39b3346e92cb94cd362a78dd83c96f33.jpg temp/1761889828_327365_1390943736_3a687c89edf207d4251ac0e6ac758a2d.jpg temp/1761889828_327365_1390943733_57c554302b3e824563f80f7a15781980.jpg 120it [00:00, 385.92it/s] 200it [00:00, 537.86it/s]2025-10-31 06:52:01.666894: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-31 06:52:01.985079: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-31 06:52:02.974594: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-10-31 06:52:02.976030: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1761889828_327365_1390943729_1d6a4e5d16a088f1c28cf4b1a7819907.jpg temp/1761889828_327365_1390943704_7379d5eeccc84b07ef45fc371390d6ce.jpg temp/1761889828_327365_1390943660_0c50212cba29325c8250d4f6aced9573.jpg temp/1761889828_327365_1390943620_5c3fd6a9f4680f05efa8c71d5e95f6ed.jpg temp/1761889828_327365_1390943034_a7544a6e7825c93b91209172159e27b5.jpg temp/1761889828_327365_1390942985_92d0e305ad5a4964def42c2264b85909.jpg temp/1761889828_327365_1390942851_bc14a2653a07f6dbc3af50e66c8d3e32.jpg temp/1761889828_327365_1390942847_c1eccf71deebecec3148d2c3bf95a0cf.jpg temp/1761889828_327365_1390942836_b286d37fa67e525e24c3ae5bcd391d19.jpg temp/1761889828_327365_1390942830_e3fb1f473b9671e77394325b7f7504bf.jpg temp/1761889828_327365_1390942457_7b997c98360434787c5f640fe0017c5b.jpg temp/1761889828_327365_1390942296_0f2b38007c26da0564170255f03d76dd.jpg temp/1761889828_327365_1390941218_e4c4d9db7638c85d4c51567251827fbf.jpg temp/1761889828_327365_1390941170_c22e17a50fa3725946ea7129285e6527.jpg temp/1761889828_327365_1390941124_594ab96ac635cdf492b80e0da7cb27d6.jpg temp/1761889828_327365_1390941085_d1b4a3650a8b1ede087e429aad59deee.jpg temp/1761889828_327365_1390941042_fb1fb34e8e5c271b2ab00129c33bc580.jpg temp/1761889828_327365_1390940984_99c16fd9606441851c88154c7b91f2b8.jpg temp/1761889828_327365_1390940164_f0a21b02544e7adf05cee3ff3a9102ae.jpg temp/1761889828_327365_1390940159_c5eab6ac93eb904664592f356db8555e.jpg temp/1761889828_327365_1390940156_43b6d65efaef50617ad95e7ad7e72d32.jpg temp/1761889828_327365_1390940152_8a210fe974757db06082a357b6e2747a.jpg temp/1761889828_327365_1390940149_3824ad29ea3382a6c468e96ec5499312.jpg temp/1761889828_327365_1390940144_c331f39079a3cb6078f68ade55780c4b.jpg temp/1761889828_327365_1390939771_87acf6147e189aa4f53cc5e224a4e7f6.jpg temp/1761889828_327365_1390939767_df3fa6e116d7e5fd154496508d4153a4.jpg temp/1761889828_327365_1390939729_9bb7c0d1dcd77df4ebfa8a6385b8a19c.jpg temp/1761889828_327365_1390939728_019ab943d068ac4f501d2bd666530770.jpg temp/1761889828_327365_1390939705_217933b5c88d07d7a8c61dcfc2bcd2e7.jpg temp/1761889828_327365_1390939555_166c394ad8e3a9e88e0c18edec409787.jpg temp/1761889828_327365_1390939381_17a14e81753c5af0a3a97ce2cf965957.jpg temp/1761889828_327365_1390937708_5bd95e1137039d2a812509987328bef8.jpg temp/1761889828_327365_1390937704_58582bf10f2f9dde30cbee51d279d4ba.jpg temp/1761889828_327365_1390937702_939b48562d3e4e5e3aeec96e9898f958.jpg temp/1761889828_327365_1390937699_009bd476000ef5414253a8892fecff22.jpg temp/1761889828_327365_1390937696_80601c6a3805813104a3b1a222786bf8.jpg temp/1761889828_327365_1390937693_551db9ae5431e93e9984657a3a8fe1ca.jpg temp/1761889828_327365_1390937438_efcb02a8dcef5d6a7bd7c32b6105aa5a.jpg temp/1761889828_327365_1390937426_777c93daba6290683c18ba2693278ab7.jpg temp/1761889828_327365_1390937423_3c2dc7e762086de9a8ecbf174c688318.jpg temp/1761889828_327365_1390937420_1b4261450b64eae713be6e700d29816f.jpg temp/1761889828_327365_1390937400_3a6d5edd52a44b9077a31c9d75519bc0.jpg temp/1761889828_327365_1390937357_7f88e1fb8342dbf1287a20e3fc5a3e8c.jpg temp/1761889828_327365_1390936647_0c4ac8b4f2d374f5fe007cb261e2ba7f.jpg temp/1761889828_327365_1390936586_134348e073edfd0e17bb86aebe034a65.jpg temp/1761889828_327365_1390936539_2a6be9e386d0cf9b2f694a6451c2f3ba.jpg temp/1761889828_327365_1390936493_7e73c20de16a536e9e9c262a91aee71f.jpg temp/1761889828_327365_1390936446_ef8e865d2aa7436379ba96d2a2f273f0.jpg temp/1761889828_327365_1390936393_5f5a6cb398d4a36b280710193c261e7b.jpg temp/1761889828_327365_1390935940_b3241b000f18561609d1fe5c8ebfef76.jpg temp/1761889828_327365_1390935891_d1cef6328659f4c445d1ce55dfb1a498.jpg temp/1761889828_327365_1390935888_e0398b8d581cdb914deaf28e42010b87.jpg temp/1761889828_327365_1390935855_39e5af3d313f979484f39b34af8e1cc2.jpg temp/1761889828_327365_1390935801_43ca89c76365e7fcd056db6330072475.jpg temp/1761889828_327365_1390935578_1626dbe6f406568fc3eb229b866d9048.jpg temp/1761889828_327365_1390934850_1f6fbe55fb6e017189cf5875983192de.jpg temp/1761889828_327365_1390934652_304ff1dae0bc943a84ed8396d0766c06.jpg temp/1761889828_327365_1390934417_5515c560fbdc58dd73a222e11cb5cfcb.jpg temp/1761889828_327365_1390934314_db5a10fca5c0f0f0428c8f645c1c1be5.jpg temp/1761889828_327365_1390934311_ff47092d57c838dbfa5cea63a5df52bd.jpg temp/1761889828_327365_1390934307_c7592a768d6d1ab80b3e22ba0ecb1df1.jpg temp/1761889828_327365_1390932018_508ff0b647ee7216b6fc656125399528.jpg temp/1761889828_327365_1390932017_dd2c4bae66c562885a217596a81fa77f.jpg temp/1761889828_327365_1390932016_250e6850286e5976d5aa7cfa5223ff0b.jpg temp/1761889828_327365_1390932012_5ecb31a9b828d621c37d2db20d0bfc0f.jpg temp/1761889828_327365_1390932009_765125f6b64d25aeb637f8329954a5b7.jpg temp/1761889828_327365_1390931982_6541ee06bd62e957695bea32f27d44c1.jpg temp/1761889828_327365_1390931807_10e8ea91eaa56320a05a563350d54874.jpg temp/1761889828_327365_1390930637_8c7f400f1c62b65b8f2a922a6ee02b77.jpg temp/1761889828_327365_1390930633_f2415e6595033b9e3b9bcd7665fc3066.jpg temp/1761889828_327365_1390930630_48e1fd4abb34339c2c873a3e7ddfe959.jpg temp/1761889828_327365_1390930626_e7280b3e46b2bedfde088954a2303b40.jpg temp/1761889828_327365_1390930607_2cbf08a691c6166be03bef7e9aa31742.jpg temp/1761889828_327365_1390930565_9eb91779520350c0b3a38bdeb748cdb2.jpg temp/1761889828_327365_1390928053_bbd883ef67840b090a5cbe2c739cc2c0.jpg temp/1761889828_327365_1390928024_f0a472f852623a51af8d7859135d7869.jpg temp/1761889828_327365_1390925257_73d5ff1eab567bf1c50328fcd0a8dd27.jpg temp/1761889828_327365_1390925208_9162961b140490973159301340034b77.jpg temp/1761889828_327365_1390925168_b788957128c9e401e7d5ed4fef7b8c8f.jpg temp/1761889828_327365_1390925125_650552a5c5136eab43282e0d727c82c6.jpg temp/1761889828_327365_1390925083_8d8f21c20ad67ec1782755b41f69e650.jpg temp/1761889828_327365_1390925044_1d0855190bd3a7477684ce713c1a0060.jpg temp/1761889828_327365_1390924752_b60c588a919a4af4913b7a2e88621d4b.jpg temp/1761889828_327365_1390924591_91556df77f3c201783eaa75dcd8b8648.jpg temp/1761889828_327365_1390922937_e29a382fcf7de3d4896d84cd55c5e989.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2339, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2533, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1390971625, 1390971418, 1390971202, 1390970993, 1390970915, 1390970869, 1390970532, 1390970523, 1390970505, 1390970501, 1390970498, 1390970495, 1390970236, 1390970213, 1390970210, 1390970208, 1390969630, 1390969589, 1390969546, 1390969504, 1390969463, 1390969420, 1390968831, 1390968788, 1390968735, 1390968723, 1390968718, 1390968714, 1390966694, 1390966654, 1390966579, 1390966538, 1390966447, 1390966400, 1390964239, 1390964236, 1390964231, 1390964217, 1390964176, 1390964105, 1390958379, 1390958353, 1390958289, 1390958231, 1390958187, 1390958146, 1390956694, 1390956677, 1390956501, 1390956458, 1390956424, 1390956413, 1390956409, 1390956374, 1390956328, 1390956285, 1390956245, 1390956202, 1390956163, 1390955285, 1390955280, 1390955269, 1390955261, 1390955257, 1390955253, 1390954269, 1390953498, 1390953454, 1390953408, 1390953380, 1390953378, 1390953374, 1390952946, 1390952929, 1390952868, 1390952843, 1390952840, 1390952818, 1390952748, 1390952699, 1390950658, 1390950656, 1390950646, 1390950634, 1390950596, 1390950556, 1390949204, 1390949198, 1390949195, 1390949192, 1390949188, 1390949175, 1390948461, 1390948415, 1390948364, 1390948320, 1390948273, 1390948227, 1390947100, 1390947032, 1390944762, 1390944716, 1390944669, 1390944613, 1390944571, 1390944565, 1390944518, 1390944487, 1390944483, 1390944478, 1390944475, 1390944472, 1390944443, 1390943736, 1390943733, 1390943729, 1390943704, 1390943660, 1390943620, 1390943034, 1390942985, 1390942851, 1390942847, 1390942836, 1390942830, 1390942457, 1390942296, 1390941218, 1390941170, 1390941124, 1390941085, 1390941042, 1390940984, 1390940164, 1390940159, 1390940156, 1390940152, 1390940149, 1390940144, 1390939771, 1390939767, 1390939729, 1390939728, 1390939705, 1390939555, 1390939381, 1390937708, 1390937704, 1390937702, 1390937699, 1390937696, 1390937693, 1390937438, 1390937426, 1390937423, 1390937420, 1390937400, 1390937357, 1390936647, 1390936586, 1390936539, 1390936493, 1390936446, 1390936393, 1390935940, 1390935891, 1390935888, 1390935855, 1390935801, 1390935578, 1390934850, 1390934652, 1390934417, 1390934314, 1390934311, 1390934307, 1390932018, 1390932017, 1390932016, 1390932012, 1390932009, 1390931982, 1390931807, 1390930637, 1390930633, 1390930630, 1390930626, 1390930607, 1390930565, 1390928053, 1390928024, 1390925257, 1390925208, 1390925168, 1390925125, 1390925083, 1390925044, 1390924752, 1390924591, 1390922937] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.193007230758667 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 58.0702702999115 time spend to save output : 0.20128464698791504 total time spend for step 0 : 58.271554946899414 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 15.34user 4.93system 1:39.29elapsed 20%CPU (0avgtext+0avgdata 2769320maxresident)k 583448inputs+513720outputs (646major+879070minor)pagefaults 0swaps