python /home/admin/mtr/script_for_cron.py -j default -m 10 -a 'python3 ~/workarea/git/Velours/python/mtr/mask_rcnn/mask_rcnn_train.py --name=learn_mask_pancarte_230525 --hashtag_type=5007 -P 23305268 --class_name=badge,pancarte --steps_per_epoch=40 --lr=0.0005 ' -s learn_mask -M 31 -S 0 -U 100,100,120 2025-05-23 17:48:30.423776: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-23 17:48:30.459157: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-23 17:48:30.461459: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fabc8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-23 17:48:30.461505: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-23 17:48:30.466306: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-23 17:48:30.754845: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1af213b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-23 17:48:30.754888: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-23 17:48:30.756685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-23 17:48:30.758264: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-23 17:48:30.790533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-23 17:48:30.808937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-23 17:48:30.813102: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-23 17:48:30.844610: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-23 17:48:30.849424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-23 17:48:30.905541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-23 17:48:30.908034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-23 17:48:30.908469: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-23 17:48:30.910058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-23 17:48:30.910077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-23 17:48:30.910087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-23 17:48:30.912092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_rcnn_train.py:33: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. model_param file didn't exist model_name : mask_coco_origin model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : [] file manque in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] local folder : /data/models_weight/mask_coco_origin import MySQLdb succeeded Import error (python version) warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.0005 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_mask_pancarte_230525 NUM_CLASSES 3 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 40 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 100 WEIGHT_DECAY 0.0001 launch train use defaut coco pre_trained model : /data/models_weight/mask_coco_origin/mask_model.h5 492993645 496005930 batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 2 chid ids of type : 5007 ++batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +batch 1 Loaded 1 chid ids of type : 5007 +{492993645: 'badge', 496005930: 'pancarte'} badge 492993645 1356331534 x0 : 639 y1 : 2294 width : 2076, height : 1311, area : 2721636, score : 1.0 None pancarte 496005930 1356331534 x0 : -1 y1 : 4158 width : 3125, height : 2735, area : 8546875, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331534.jpg pancarte 496005930 1356331538 x0 : 1319 y1 : 2643 width : 1206, height : 826, area : 996156, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331538.jpg pancarte 496005930 1356331545 x0 : 1109 y1 : 1843 width : 546, height : 431, area : 235326, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331545.jpg pancarte 496005930 1356331553 x0 : 971 y1 : 1646 width : 888, height : 679, area : 602952, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331553.jpg pancarte 496005930 1356331557 x0 : 1707 y1 : 1808 width : 774, height : 617, area : 477558, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331557.jpg pancarte 496005930 1356331559 x0 : 2057 y1 : 1619 width : 670, height : 514, area : 344380, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331559.jpg pancarte 496005930 1356331574 x0 : 0 y1 : 4159 width : 3119, height : 3332, area : 10392508, score : 1.0 None badge 492993645 1356331574 x0 : 772 y1 : 2309 width : 2341, height : 1582, area : 3703462, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331574.jpg pancarte 496005930 1356331578 x0 : 1710 y1 : 2479 width : 1107, height : 869, area : 961983, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331578.jpg pancarte 496005930 1356331586 x0 : 768 y1 : 2082 width : 840, height : 639, area : 536760, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331586.jpg pancarte 496005930 1356331593 x0 : 1150 y1 : 2168 width : 994, height : 818, area : 813092, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331593.jpg pancarte 496005930 1356331598 x0 : 1303 y1 : 2819 width : 1093, height : 982, area : 1073326, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331598.jpg pancarte 496005930 1356331602 x0 : 1052 y1 : 2716 width : 1124, height : 996, area : 1119504, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331602.jpg pancarte 496005930 1356331619 x0 : 2024 y1 : 2197 width : 725, height : 604, area : 437900, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331619.jpg badge 492993645 1356331626 x0 : 2158 y1 : 1396 width : 661, height : 455, area : 300755, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331626.jpg badge 492993645 1356331643 x0 : 1799 y1 : 1208 width : 976, height : 652, area : 636352, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331643.jpg pancarte 496005930 1356331654 x0 : 1386 y1 : 2579 width : 963, height : 832, area : 801216, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331654.jpg pancarte 496005930 1356331661 x0 : 1304 y1 : 2433 width : 896, height : 665, area : 595840, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331661.jpg badge 492993645 1356331664 x0 : 390 y1 : 3823 width : 2211, height : 3267, area : 7223337, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331664.jpg pancarte 496005930 1356331673 x0 : 1495 y1 : 1563 width : 1086, height : 723, area : 785178, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331673.jpg pancarte 496005930 1356331680 x0 : 2210 y1 : 1252 width : 302, height : 241, area : 72782, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331680.jpg pancarte 496005930 1356331687 x0 : 2909 y1 : 1573 width : 847, height : 607, area : 514129, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331687.jpg pancarte 496005930 1356331694 x0 : 2557 y1 : 1303 width : 360, height : 309, area : 111240, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331694.jpg pancarte 496005930 1356331701 x0 : 1174 y1 : 2191 width : 466, height : 316, area : 147256, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331701.jpg pancarte 496005930 1356331706 x0 : 696 y1 : 2860 width : 652, height : 526, area : 342952, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331706.jpg pancarte 496005930 1356331714 x0 : -1 y1 : 1164 width : 1367, height : 1164, area : 1591188, score : 1.0 None badge 492993645 1356331714 x0 : 0 y1 : 3010 width : 3124, height : 1839, area : 5745036, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331714.jpg pancarte 496005930 1356331719 x0 : 1962 y1 : 2387 width : 1408, height : 840, area : 1182720, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331719.jpg pancarte 496005930 1356331730 x0 : 1776 y1 : 1579 width : 666, height : 552, area : 367632, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331730.jpg pancarte 496005930 1356331735 x0 : 1135 y1 : 1927 width : 1195, height : 912, area : 1089840, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331735.jpg pancarte 496005930 1356331742 x0 : 2158 y1 : 1713 width : 950, height : 790, area : 750500, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331742.jpg pancarte 496005930 1356331746 x0 : 1992 y1 : 1564 width : 1138, height : 945, area : 1075410, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331746.jpg pancarte 496005930 1356331757 x0 : 1927 y1 : 1204 width : 823, height : 659, area : 542357, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331757.jpg pancarte 496005930 1356331762 x0 : 2647 y1 : 1081 width : 1122, height : 836, area : 937992, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331762.jpg pancarte 496005930 1356331766 x0 : 0 y1 : 3120 width : 4164, height : 3120, area : 12991680, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331766.jpg badge 492993645 1356331771 x0 : 0 y1 : 4159 width : 3122, height : 2498, area : 7798756, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331771.jpg pancarte 496005930 1356331779 x0 : 2154 y1 : 2091 width : 690, height : 632, area : 436080, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331779.jpg pancarte 496005930 1356331782 x0 : 1299 y1 : 1413 width : 1255, height : 826, area : 1036630, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331782.jpg pancarte 496005930 1356331788 x0 : 1995 y1 : 2575 width : 1505, height : 952, area : 1432760, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331788.jpg pancarte 496005930 1356331794 x0 : 1898 y1 : 2201 width : 793, height : 693, area : 549549, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331794.jpg pancarte 496005930 1356331801 x0 : 1475 y1 : 2996 width : 1319, height : 960, area : 1266240, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331801.jpg pancarte 496005930 1356331809 x0 : 2090 y1 : 1655 width : 614, height : 422, area : 259108, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331809.jpg pancarte 496005930 1356331815 x0 : 2211 y1 : 1627 width : 920, height : 670, area : 616400, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331815.jpg pancarte 496005930 1356331817 x0 : 1719 y1 : 2476 width : 795, height : 597, area : 474615, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331817.jpg badge 492993645 1356331838 x0 : 1 y1 : 3218 width : 3117, height : 2003, area : 6243351, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331838.jpg pancarte 496005930 1356331842 x0 : 1899 y1 : 2144 width : 1004, height : 771, area : 774084, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331842.jpg pancarte 496005930 1356331848 x0 : 2100 y1 : 2105 width : 1036, height : 806, area : 835016, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331848.jpg pancarte 496005930 1356331857 x0 : 2091 y1 : 2548 width : 1358, height : 999, area : 1356642, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331857.jpg pancarte 496005930 1356331862 x0 : 1590 y1 : 1690 width : 692, height : 396, area : 274032, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331862.jpg pancarte 496005930 1356331867 x0 : 1746 y1 : 1325 width : 845, height : 498, area : 420810, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331867.jpg pancarte 496005930 1356331876 x0 : 1609 y1 : 1724 width : 663, height : 474, area : 314262, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331876.jpg pancarte 496005930 1356331884 x0 : 1702 y1 : 2140 width : 1054, height : 767, area : 808418, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331884.jpg pancarte 496005930 1356331894 x0 : 172 y1 : 3120 width : 1632, height : 1175, area : 1917600, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331894.jpg pancarte 496005930 1356331903 x0 : 942 y1 : 2827 width : 989, height : 886, area : 876254, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331903.jpg pancarte 496005930 1356331907 x0 : 2192 y1 : 2257 width : 1537, height : 896, area : 1377152, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331907.jpg badge 492993645 1356331909 x0 : 1047 y1 : 2216 width : 3004, height : 1739, area : 5223956, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331909.jpg pancarte 496005930 1356331918 x0 : 1770 y1 : 2436 width : 632, height : 525, area : 331800, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331918.jpg pancarte 496005930 1356331923 x0 : 1454 y1 : 2979 width : 1160, height : 999, area : 1158840, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331923.jpg pancarte 496005930 1356331931 x0 : 1854 y1 : 1823 width : 484, height : 392, area : 189728, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331931.jpg pancarte 496005930 1356331943 x0 : 1826 y1 : 1571 width : 738, height : 572, area : 422136, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331943.jpg pancarte 496005930 1356331946 x0 : 2088 y1 : 1812 width : 782, height : 635, area : 496570, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331946.jpg pancarte 496005930 1356331957 x0 : 1804 y1 : 2287 width : 1318, height : 770, area : 1014860, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331957.jpg badge 492993645 1356331962 x0 : 9 y1 : 2615 width : 4151, height : 2608, area : 10825808, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331962.jpg pancarte 496005930 1356331967 x0 : 1452 y1 : 2869 width : 1218, height : 1081, area : 1316658, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331967.jpg pancarte 496005930 1356331976 x0 : 1331 y1 : 1960 width : 676, height : 511, area : 345436, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331976.jpg pancarte 496005930 1356331982 x0 : 1110 y1 : 2192 width : 693, height : 589, area : 408177, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331982.jpg pancarte 496005930 1356331985 x0 : 1999 y1 : 1817 width : 604, height : 480, area : 289920, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331985.jpg pancarte 496005930 1356331995 x0 : 1953 y1 : 2553 width : 1224, height : 817, area : 1000008, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356331995.jpg badge 492993645 1356332003 x0 : 2 y1 : 2685 width : 4149, height : 2685, area : 11140065, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356332003.jpg pancarte 496005930 1356332010 x0 : 2111 y1 : 2391 width : 861, height : 606, area : 521766, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356332010.jpg pancarte 496005930 1356332017 x0 : 2273 y1 : 2433 width : 939, height : 678, area : 636642, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356332017.jpg pancarte 496005930 1356332023 x0 : 1082 y1 : 1448 width : 366, height : 283, area : 103578, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356332023.jpg pancarte 496005930 1356332025 x0 : 2597 y1 : 2307 width : 504, height : 1056, area : 532224, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356332025.jpg pancarte 496005930 1356332042 x0 : 1872 y1 : 2500 width : 771, height : 549, area : 423279, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356332042.jpg pancarte 496005930 1356361517 x0 : 2274 y1 : 2363 width : 1381, height : 872, area : 1204232, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356361517.jpg badge 492993645 1356361523 x0 : 2010 y1 : 1624 width : 712, height : 445, area : 316840, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356361523.jpg pancarte 496005930 1356361530 x0 : 1595 y1 : 2256 width : 729, height : 597, area : 435213, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356361530.jpg pancarte 496005930 1356361537 x0 : 1900 y1 : 2125 width : 903, height : 724, area : 653772, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356361537.jpg pancarte 496005930 1356361543 x0 : 2208 y1 : 2389 width : 1015, height : 858, area : 870870, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356361543.jpg pancarte 496005930 1356361548 x0 : 1707 y1 : 2804 width : 1011, height : 880, area : 889680, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356361548.jpg pancarte 496005930 1356982967 x0 : 1773 y1 : 2444 width : 1347, height : 815, area : 1097805, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356982967.jpg badge 492993645 1356982971 x0 : 710 y1 : 4147 width : 2395, height : 3474, area : 8320230, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356982971.jpg pancarte 496005930 1356982975 x0 : 2092 y1 : 2226 width : 630, height : 502, area : 316260, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356982975.jpg pancarte 496005930 1356982983 x0 : 2134 y1 : 2335 width : 850, height : 675, area : 573750, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356982983.jpg pancarte 496005930 1356982991 x0 : 2351 y1 : 2008 width : 659, height : 532, area : 350588, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356982991.jpg pancarte 496005930 1356982997 x0 : 1309 y1 : 2176 width : 682, height : 510, area : 347820, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356982997.jpg pancarte 496005930 1356983040 x0 : 2101 y1 : 2457 width : 1047, height : 834, area : 873198, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356983040.jpg pancarte 496005930 1356983046 x0 : 1961 y1 : 2890 width : 1298, height : 1056, area : 1370688, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356983046.jpg badge 492993645 1356983052 x0 : 697 y1 : 2314 width : 2560, height : 1594, area : 4080640, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356983052.jpg pancarte 496005930 1356983061 x0 : 1541 y1 : 2587 width : 717, height : 552, area : 395784, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356983061.jpg pancarte 496005930 1356983067 x0 : 1264 y1 : 2104 width : 720, height : 543, area : 390960, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356983067.jpg pancarte 496005930 1356983071 x0 : 1181 y1 : 1546 width : 441, height : 298, area : 131418, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356983071.jpg pancarte 496005930 1356983086 x0 : 1603 y1 : 2607 width : 739, height : 540, area : 399060, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356983086.jpg pancarte 496005930 1356998184 x0 : 2033 y1 : 2402 width : 1383, height : 930, area : 1286190, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998184.jpg pancarte 496005930 1356998222 x0 : 2246 y1 : 2293 width : 635, height : 531, area : 337185, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998222.jpg pancarte 496005930 1356998225 x0 : 1619 y1 : 3994 width : 1188, height : 897, area : 1065636, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998225.jpg pancarte 496005930 1356998228 x0 : 1872 y1 : 2231 width : 671, height : 483, area : 324093, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998228.jpg pancarte 496005930 1356998234 x0 : 1252 y1 : 2346 width : 537, height : 290, area : 155730, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998234.jpg pancarte 496005930 1356998239 x0 : 1994 y1 : 1998 width : 515, height : 395, area : 203425, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998239.jpg pancarte 496005930 1356998244 x0 : 932 y1 : 2481 width : 665, height : 519, area : 345135, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998244.jpg pancarte 496005930 1356998320 x0 : 1577 y1 : 1513 width : 269, height : 241, area : 64829, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998320.jpg pancarte 496005930 1356998437 x0 : 1634 y1 : 3112 width : 950, height : 852, area : 809400, score : 1.0 None pancarte 496005930 1356998437 x0 : 1633 y1 : 3108 width : 945, height : 849, area : 802305, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998437.jpg pancarte 496005930 1356998545 x0 : 1214 y1 : 1860 width : 401, height : 363, area : 145563, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998545.jpg pancarte 496005930 1356998665 x0 : 1165 y1 : 2263 width : 904, height : 650, area : 587600, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998665.jpg badge 492993645 1356998732 x0 : 409 y1 : 4159 width : 2665, height : 4065, area : 10833225, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/train/1356998732.jpg pancarte 496005930 1356998753 Using TensorFlow backend. WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1659: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. 2025-05-23 17:49:34.831724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-23 17:49:34.832173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-23 17:49:34.832203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-23 17:49:34.832231: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-23 17:49:34.832254: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-23 17:49:34.832271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-23 17:49:34.832286: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-23 17:49:34.832303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-23 17:49:34.833671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:570: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead 2025-05-23 17:49:40.909704: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 17:49:40.910176: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1363] Profiler found 1 GPUs 2025-05-23 17:49:40.917619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcupti.so.10.1 2025-05-23 17:49:41.006718: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:49:45.802721: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-23 17:49:45.802807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-23 17:49:45.802827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-23 17:49:45.802844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-23 17:49:45.802860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-23 17:49:45.802877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-23 17:49:45.802892: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-23 17:49:45.802922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-23 17:49:45.804512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-23 17:49:45.804562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-23 17:49:45.804573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-23 17:49:45.804583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-23 17:49:45.806744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/engine/training_generator.py:48: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class. warnings.warn( WARNING:tensorflow:Model failed to serialize as JSON. Ignoring... cannot pickle '_thread.RLock' object x0 : 2109 y1 : 2432 width : 1295, height : 864, area : 1118880, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1356998753.jpg pancarte 496005930 1356998806 x0 : 1661 y1 : 1549 width : 608, height : 523, area : 317984, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1356998806.jpg pancarte 496005930 1356998813 x0 : 899 y1 : 1307 width : 1136, height : 1121, area : 1273456, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1356998813.jpg pancarte 496005930 1356998821 x0 : 1208 y1 : 1234 width : 482, height : 388, area : 187016, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1356998821.jpg pancarte 496005930 1356998826 x0 : 1050 y1 : 1365 width : 389, height : 322, area : 125258, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1356998826.jpg pancarte 496005930 1356998832 x0 : 1729 y1 : 1676 width : 470, height : 411, area : 193170, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1356998832.jpg pancarte 496005930 1357676318 x0 : 1942 y1 : 2041 width : 1334, height : 816, area : 1088544, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1357676318.jpg pancarte 496005930 1357676324 x0 : 1992 y1 : 2636 width : 928, height : 643, area : 596704, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1357676324.jpg pancarte 496005930 1357676331 x0 : 1758 y1 : 2800 width : 1140, height : 797, area : 908580, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1357676331.jpg pancarte 496005930 1357676341 x0 : 1756 y1 : 2940 width : 888, height : 948, area : 841824, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1357676341.jpg pancarte 496005930 1357676347 x0 : 1714 y1 : 3069 width : 1170, height : 1037, area : 1213290, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1357676347.jpg pancarte 496005930 1357676358 x0 : 1940 y1 : 2721 width : 882, height : 676, area : 596232, score : 1.0 None /data/data_root/learn_mask_pancarte_230525/val/1357676358.jpg number of images :103 number of images :12 Training network heads Starting at epoch 0. LR=0.0005 Checkpoint Path: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/mask_rcnn_learn_mask_pancarte_230525_{epoch:04d}.h5 Selecting layers to train fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) 2025-05-23 17:50:23.223510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-23 17:50:23.933696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 Epoch 1/2 1/40 [..............................] - ETA: 12:29 - loss: 5.7512 - rpn_class_loss: 4.0106e-04 - rpn_bbox_loss: 2.2052 - mrcnn_class_loss: 0.4524 - mrcnn_bbox_loss: 1.7448 - mrcnn_mask_loss: 1.34852025-05-23 17:50:31.125511: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 6:27 - loss: 5.6746 - rpn_class_loss: 4.0003e-04 - rpn_bbox_loss: 2.1897 - mrcnn_class_loss: 0.4231 - mrcnn_bbox_loss: 1.7334 - mrcnn_mask_loss: 1.3280 2025-05-23 17:50:31.389098: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:31.390698: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3439 callback api events and 3439 activity events. 2025-05-23 17:50:31.611817: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_31 2025-05-23 17:50:31.704496: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_31/marlene.trace.json.gz 2025-05-23 17:50:31.788386: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.477 ms 2025-05-23 17:50:31.820430: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_31Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_31/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_31/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_31/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_31/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 4:19 - loss: 4.2308 - rpn_class_loss: 0.0296 - rpn_bbox_loss: 1.8348 - mrcnn_class_loss: 0.3254 - mrcnn_bbox_loss: 1.1556 - mrcnn_mask_loss: 0.8854 2025-05-23 17:50:32.053558: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 3:11 - loss: 4.0745 - rpn_class_loss: 0.0269 - rpn_bbox_loss: 1.4136 - mrcnn_class_loss: 0.3118 - mrcnn_bbox_loss: 1.1875 - mrcnn_mask_loss: 1.13462025-05-23 17:50:32.289242: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:32.289993: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:50:32.449129: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_32 2025-05-23 17:50:32.516821: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_32/marlene.trace.json.gz 2025-05-23 17:50:32.565142: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.305 ms 2025-05-23 17:50:32.599021: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_32Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_32/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_32/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_32/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_32/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 2:33 - loss: 3.9885 - rpn_class_loss: 0.0479 - rpn_bbox_loss: 1.2925 - mrcnn_class_loss: 0.2633 - mrcnn_bbox_loss: 1.2762 - mrcnn_mask_loss: 1.10852025-05-23 17:50:32.887056: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 2:05 - loss: 3.4791 - rpn_class_loss: 0.0585 - rpn_bbox_loss: 1.2121 - mrcnn_class_loss: 0.2212 - mrcnn_bbox_loss: 1.0635 - mrcnn_mask_loss: 0.92382025-05-23 17:50:33.150133: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:33.151135: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3255 callback api events and 3255 activity events. 2025-05-23 17:50:33.332415: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_33 2025-05-23 17:50:33.420025: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_33/marlene.trace.json.gz 2025-05-23 17:50:33.486232: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.481 ms 2025-05-23 17:50:33.524216: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_33Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_33/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_33/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_33/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_33/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 1:47 - loss: 3.5568 - rpn_class_loss: 0.0533 - rpn_bbox_loss: 1.1253 - mrcnn_class_loss: 0.2171 - mrcnn_bbox_loss: 1.0950 - mrcnn_mask_loss: 1.06622025-05-23 17:50:33.796662: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 1:32 - loss: 3.5478 - rpn_class_loss: 0.0535 - rpn_bbox_loss: 1.0838 - mrcnn_class_loss: 0.2005 - mrcnn_bbox_loss: 1.1657 - mrcnn_mask_loss: 1.04432025-05-23 17:50:34.060273: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:34.061566: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3361 callback api events and 3361 activity events. 2025-05-23 17:50:34.235722: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_34 2025-05-23 17:50:34.326399: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_34/marlene.trace.json.gz 2025-05-23 17:50:34.390476: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.508 ms 2025-05-23 17:50:34.428903: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_34/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 1:21 - loss: 3.4972 - rpn_class_loss: 0.0572 - rpn_bbox_loss: 1.0387 - mrcnn_class_loss: 0.1855 - mrcnn_bbox_loss: 1.1959 - mrcnn_mask_loss: 1.01992025-05-23 17:50:34.675961: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 1:11 - loss: 3.2298 - rpn_class_loss: 0.0602 - rpn_bbox_loss: 1.0083 - mrcnn_class_loss: 0.1670 - mrcnn_bbox_loss: 1.0763 - mrcnn_mask_loss: 0.91792025-05-23 17:50:34.964050: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:34.964971: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3255 callback api events and 3255 activity events. 2025-05-23 17:50:35.176890: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_35 2025-05-23 17:50:35.329673: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_35/marlene.trace.json.gz 2025-05-23 17:50:35.435160: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.629 ms 2025-05-23 17:50:35.473739: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_35Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_35/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_35/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_35/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_35/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 1:05 - loss: 3.1031 - rpn_class_loss: 0.0645 - rpn_bbox_loss: 0.9413 - mrcnn_class_loss: 0.1537 - mrcnn_bbox_loss: 1.0433 - mrcnn_mask_loss: 0.90032025-05-23 17:50:35.802844: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 58s - loss: 3.1100 - rpn_class_loss: 0.0594 - rpn_bbox_loss: 0.9501 - mrcnn_class_loss: 0.1494 - mrcnn_bbox_loss: 1.0392 - mrcnn_mask_loss: 0.9120 2025-05-23 17:50:36.142951: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:36.144233: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3439 callback api events and 3439 activity events. 2025-05-23 17:50:36.386630: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_36 2025-05-23 17:50:36.487328: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_36/marlene.trace.json.gz 2025-05-23 17:50:36.576418: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.682 ms 2025-05-23 17:50:36.617249: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_36Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_36/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_36/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_36/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_36/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 53s - loss: 2.9303 - rpn_class_loss: 0.0602 - rpn_bbox_loss: 0.9312 - mrcnn_class_loss: 0.1379 - mrcnn_bbox_loss: 0.9593 - mrcnn_mask_loss: 0.84182025-05-23 17:50:36.897023: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 1:28 - loss: 2.7710 - rpn_class_loss: 0.0585 - rpn_bbox_loss: 0.9120 - mrcnn_class_loss: 0.1281 - mrcnn_bbox_loss: 0.8907 - mrcnn_mask_loss: 0.78172025-05-23 17:50:58.551094: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:58.552126: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3255 callback api events and 3255 activity events. 2025-05-23 17:50:58.807796: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_58 2025-05-23 17:50:58.908354: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_58/marlene.trace.json.gz 2025-05-23 17:50:58.971318: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.375 ms 2025-05-23 17:50:59.007979: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_58Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_58/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_58/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_58/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_58/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 1:20 - loss: 2.7122 - rpn_class_loss: 0.0558 - rpn_bbox_loss: 0.8563 - mrcnn_class_loss: 0.1325 - mrcnn_bbox_loss: 0.8850 - mrcnn_mask_loss: 0.78262025-05-23 17:50:59.298352: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 1:12 - loss: 2.6597 - rpn_class_loss: 0.0579 - rpn_bbox_loss: 0.8406 - mrcnn_class_loss: 0.1285 - mrcnn_bbox_loss: 0.8622 - mrcnn_mask_loss: 0.77052025-05-23 17:50:59.607716: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:50:59.609019: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3400 callback api events and 3400 activity events. 2025-05-23 17:50:59.865826: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_59 2025-05-23 17:50:59.970296: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_59/marlene.trace.json.gz 2025-05-23 17:51:00.060189: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.263 ms 2025-05-23 17:51:00.105852: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_59Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_59/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_59/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_59/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_50_59/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 1:06 - loss: 2.5359 - rpn_class_loss: 0.0589 - rpn_bbox_loss: 0.8193 - mrcnn_class_loss: 0.1210 - mrcnn_bbox_loss: 0.8115 - mrcnn_mask_loss: 0.72522025-05-23 17:51:00.377770: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 1:00 - loss: 2.4957 - rpn_class_loss: 0.0568 - rpn_bbox_loss: 0.7977 - mrcnn_class_loss: 0.1209 - mrcnn_bbox_loss: 0.8048 - mrcnn_mask_loss: 0.71542025-05-23 17:51:00.681241: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:00.682744: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:51:00.919361: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_00 2025-05-23 17:51:01.029140: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_00/marlene.trace.json.gz 2025-05-23 17:51:01.128285: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.537 ms 2025-05-23 17:51:01.178492: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_00/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 55s - loss: 2.5009 - rpn_class_loss: 0.0557 - rpn_bbox_loss: 0.7968 - mrcnn_class_loss: 0.1188 - mrcnn_bbox_loss: 0.8178 - mrcnn_mask_loss: 0.7119 2025-05-23 17:51:01.502594: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 50s - loss: 2.4554 - rpn_class_loss: 0.0555 - rpn_bbox_loss: 0.7896 - mrcnn_class_loss: 0.1165 - mrcnn_bbox_loss: 0.7927 - mrcnn_mask_loss: 0.70112025-05-23 17:51:01.835613: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:01.836285: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3362 callback api events and 3362 activity events. 2025-05-23 17:51:02.033929: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_01 2025-05-23 17:51:02.120422: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_01/marlene.trace.json.gz 2025-05-23 17:51:02.197916: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.818 ms 2025-05-23 17:51:02.239418: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_01Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_01/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_01/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_01/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_01/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 46s - loss: 2.4049 - rpn_class_loss: 0.0553 - rpn_bbox_loss: 0.7563 - mrcnn_class_loss: 0.1114 - mrcnn_bbox_loss: 0.7958 - mrcnn_mask_loss: 0.68602025-05-23 17:51:02.506634: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 42s - loss: 2.3369 - rpn_class_loss: 0.0557 - rpn_bbox_loss: 0.7350 - mrcnn_class_loss: 0.1071 - mrcnn_bbox_loss: 0.7666 - mrcnn_mask_loss: 0.67242025-05-23 17:51:02.795199: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:02.796206: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3324 callback api events and 3324 activity events. 2025-05-23 17:51:03.140910: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_02 2025-05-23 17:51:03.375095: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_02/marlene.trace.json.gz 2025-05-23 17:51:03.506842: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.042 ms 2025-05-23 17:51:03.528688: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_02Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_02/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_02/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_02/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_02/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 39s - loss: 2.3201 - rpn_class_loss: 0.0535 - rpn_bbox_loss: 0.7318 - mrcnn_class_loss: 0.1049 - mrcnn_bbox_loss: 0.7554 - mrcnn_mask_loss: 0.67442025-05-23 17:51:03.779798: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 35s - loss: 2.2410 - rpn_class_loss: 0.0524 - rpn_bbox_loss: 0.7178 - mrcnn_class_loss: 0.1006 - mrcnn_bbox_loss: 0.7239 - mrcnn_mask_loss: 0.64632025-05-23 17:51:04.037023: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:04.037895: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3255 callback api events and 3255 activity events. 2025-05-23 17:51:04.233578: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_04 2025-05-23 17:51:04.311781: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_04/marlene.trace.json.gz 2025-05-23 17:51:04.368555: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.266 ms 2025-05-23 17:51:04.393812: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_04Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_04/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_04/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_04/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_04/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 32s - loss: 2.2371 - rpn_class_loss: 0.0505 - rpn_bbox_loss: 0.7137 - mrcnn_class_loss: 0.1044 - mrcnn_bbox_loss: 0.7330 - mrcnn_mask_loss: 0.63542025-05-23 17:51:04.622821: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 29s - loss: 2.2324 - rpn_class_loss: 0.0500 - rpn_bbox_loss: 0.7259 - mrcnn_class_loss: 0.1011 - mrcnn_bbox_loss: 0.7315 - mrcnn_mask_loss: 0.62382025-05-23 17:51:05.781996: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:05.782434: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3324 callback api events and 3324 activity events. 2025-05-23 17:51:05.904214: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_05 2025-05-23 17:51:05.968416: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_05/marlene.trace.json.gz 2025-05-23 17:51:06.023764: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.564 ms 2025-05-23 17:51:06.053242: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_05Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_05/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_05/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_05/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_05/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 27s - loss: 2.2194 - rpn_class_loss: 0.0483 - rpn_bbox_loss: 0.7221 - mrcnn_class_loss: 0.1021 - mrcnn_bbox_loss: 0.7328 - mrcnn_mask_loss: 0.61402025-05-23 17:51:08.089528: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 26s - loss: 2.2345 - rpn_class_loss: 0.0472 - rpn_bbox_loss: 0.7298 - mrcnn_class_loss: 0.1024 - mrcnn_bbox_loss: 0.7394 - mrcnn_mask_loss: 0.61582025-05-23 17:51:12.482551: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:12.484056: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3400 callback api events and 3400 activity events. 2025-05-23 17:51:12.728948: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_12 2025-05-23 17:51:12.815764: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_12/marlene.trace.json.gz 2025-05-23 17:51:12.882097: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.282 ms 2025-05-23 17:51:12.914587: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_12/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 23s - loss: 2.2030 - rpn_class_loss: 0.0477 - rpn_bbox_loss: 0.7354 - mrcnn_class_loss: 0.0991 - mrcnn_bbox_loss: 0.7173 - mrcnn_mask_loss: 0.60352025-05-23 17:51:13.153843: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 23s - loss: 2.2130 - rpn_class_loss: 0.0469 - rpn_bbox_loss: 0.7350 - mrcnn_class_loss: 0.0985 - mrcnn_bbox_loss: 0.7267 - mrcnn_mask_loss: 0.60592025-05-23 17:51:20.497485: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:20.498812: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3362 callback api events and 3362 activity events. 2025-05-23 17:51:20.764388: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_20 2025-05-23 17:51:20.872369: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_20/marlene.trace.json.gz 2025-05-23 17:51:20.934989: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 0.933 ms 2025-05-23 17:51:20.957854: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_20Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_20/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_20/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_20/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_20/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 20s - loss: 2.1878 - rpn_class_loss: 0.0460 - rpn_bbox_loss: 0.7280 - mrcnn_class_loss: 0.0964 - mrcnn_bbox_loss: 0.7229 - mrcnn_mask_loss: 0.59452025-05-23 17:51:21.172688: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 18s - loss: 2.1862 - rpn_class_loss: 0.0450 - rpn_bbox_loss: 0.7430 - mrcnn_class_loss: 0.0945 - mrcnn_bbox_loss: 0.7206 - mrcnn_mask_loss: 0.58312025-05-23 17:51:23.049225: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:23.049919: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3362 callback api events and 3362 activity events. 2025-05-23 17:51:23.199705: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_23 2025-05-23 17:51:23.268478: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_23/marlene.trace.json.gz 2025-05-23 17:51:23.315141: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.186 ms 2025-05-23 17:51:23.349972: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_23/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 15s - loss: 2.1631 - rpn_class_loss: 0.0444 - rpn_bbox_loss: 0.7472 - mrcnn_class_loss: 0.0918 - mrcnn_bbox_loss: 0.7080 - mrcnn_mask_loss: 0.57172025-05-23 17:51:26.257802: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 13s - loss: 2.1429 - rpn_class_loss: 0.0446 - rpn_bbox_loss: 0.7475 - mrcnn_class_loss: 0.0899 - mrcnn_bbox_loss: 0.7006 - mrcnn_mask_loss: 0.56022025-05-23 17:51:26.716681: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:26.717308: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3324 callback api events and 3324 activity events. 2025-05-23 17:51:27.006078: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_26 2025-05-23 17:51:27.127975: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_26/marlene.trace.json.gz 2025-05-23 17:51:27.201649: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.574 ms 2025-05-23 17:51:27.240362: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_26/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 11s - loss: 2.1280 - rpn_class_loss: 0.0436 - rpn_bbox_loss: 0.7575 - mrcnn_class_loss: 0.0884 - mrcnn_bbox_loss: 0.6884 - mrcnn_mask_loss: 0.55012025-05-23 17:51:32.009619: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 9s - loss: 2.1059 - rpn_class_loss: 0.0436 - rpn_bbox_loss: 0.7436 - mrcnn_class_loss: 0.0878 - mrcnn_bbox_loss: 0.6849 - mrcnn_mask_loss: 0.5460 2025-05-23 17:51:33.136159: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:33.136829: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3400 callback api events and 3400 activity events. 2025-05-23 17:51:33.263790: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_33 2025-05-23 17:51:33.346223: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_33/marlene.trace.json.gz 2025-05-23 17:51:33.409766: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.747 ms 2025-05-23 17:51:33.453864: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_33Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_33/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_33/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_33/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_33/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 6s - loss: 2.0801 - rpn_class_loss: 0.0437 - rpn_bbox_loss: 0.7408 - mrcnn_class_loss: 0.0860 - mrcnn_bbox_loss: 0.6737 - mrcnn_mask_loss: 0.53602025-05-23 17:51:36.318022: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 4s - loss: 2.0536 - rpn_class_loss: 0.0434 - rpn_bbox_loss: 0.7344 - mrcnn_class_loss: 0.0841 - mrcnn_bbox_loss: 0.6667 - mrcnn_mask_loss: 0.52492025-05-23 17:51:38.522348: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:38.523491: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3400 callback api events and 3400 activity events. 2025-05-23 17:51:38.861718: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_38 2025-05-23 17:51:38.946072: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_38/marlene.trace.json.gz 2025-05-23 17:51:39.031273: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.769 ms 2025-05-23 17:51:39.085644: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_38Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_38/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_38/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_38/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_38/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 2s - loss: 2.0280 - rpn_class_loss: 0.0433 - rpn_bbox_loss: 0.7202 - mrcnn_class_loss: 0.0840 - mrcnn_bbox_loss: 0.6585 - mrcnn_mask_loss: 0.52202025-05-23 17:51:41.165099: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 17:51:45.919205: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:51:45.921269: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3439 callback api events and 3439 activity events. 2025-05-23 17:51:46.190315: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_46 2025-05-23 17:51:46.330531: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_46/marlene.trace.json.gz 2025-05-23 17:51:46.427040: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.077 ms 2025-05-23 17:51:46.496180: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_46Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_46/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_46/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_46/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_51_46/marlene.kernel_stats.pb 40/40 [==============================] - 101s 3s/step - loss: 2.0658 - rpn_class_loss: 0.0426 - rpn_bbox_loss: 0.7612 - mrcnn_class_loss: 0.0866 - mrcnn_bbox_loss: 0.6629 - mrcnn_mask_loss: 0.5125 - val_loss: 0.6853 - val_rpn_class_loss: 0.0414 - val_rpn_bbox_loss: 0.1388 - val_mrcnn_class_loss: 0.0285 - val_mrcnn_bbox_loss: 0.2327 - val_mrcnn_mask_loss: 0.2439 /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Epoch 2/2 1/40 [..............................] - ETA: 16s - loss: 1.1559 - rpn_class_loss: 0.0327 - rpn_bbox_loss: 0.4820 - mrcnn_class_loss: 0.0149 - mrcnn_bbox_loss: 0.5145 - mrcnn_mask_loss: 0.11182025-05-23 17:53:22.894701: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 17s - loss: 1.9034 - rpn_class_loss: 0.0226 - rpn_bbox_loss: 0.8105 - mrcnn_class_loss: 0.1662 - mrcnn_bbox_loss: 0.6537 - mrcnn_mask_loss: 0.25042025-05-23 17:53:23.299452: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:23.307903: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3442 callback api events and 3442 activity events. 2025-05-23 17:53:23.653160: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_23 2025-05-23 17:53:23.780426: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_23/marlene.trace.json.gz 2025-05-23 17:53:23.882621: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.575 ms 2025-05-23 17:53:23.931556: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_23/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 22s - loss: 2.5528 - rpn_class_loss: 0.0233 - rpn_bbox_loss: 1.5437 - mrcnn_class_loss: 0.1715 - mrcnn_bbox_loss: 0.6037 - mrcnn_mask_loss: 0.21052025-05-23 17:53:24.213760: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 18s - loss: 1.9863 - rpn_class_loss: 0.0270 - rpn_bbox_loss: 1.2200 - mrcnn_class_loss: 0.1286 - mrcnn_bbox_loss: 0.4528 - mrcnn_mask_loss: 0.15792025-05-23 17:53:24.494067: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:24.495918: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3255 callback api events and 3255 activity events. 2025-05-23 17:53:24.850936: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_24 2025-05-23 17:53:25.036563: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_24/marlene.trace.json.gz 2025-05-23 17:53:25.223846: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.871 ms 2025-05-23 17:53:25.307525: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_24Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_24/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_24/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_24/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_24/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 22s - loss: 2.0393 - rpn_class_loss: 0.0244 - rpn_bbox_loss: 1.1300 - mrcnn_class_loss: 0.1839 - mrcnn_bbox_loss: 0.4954 - mrcnn_mask_loss: 0.20562025-05-23 17:53:25.658247: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 21s - loss: 2.1037 - rpn_class_loss: 0.0205 - rpn_bbox_loss: 1.1411 - mrcnn_class_loss: 0.1690 - mrcnn_bbox_loss: 0.5142 - mrcnn_mask_loss: 0.25902025-05-23 17:53:26.133901: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:26.140539: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3439 callback api events and 3439 activity events. 2025-05-23 17:53:26.487599: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_26 2025-05-23 17:53:26.720655: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_26/marlene.trace.json.gz 2025-05-23 17:53:26.964143: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.709 ms 2025-05-23 17:53:27.058638: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_26/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 23s - loss: 1.8327 - rpn_class_loss: 0.0228 - rpn_bbox_loss: 1.0023 - mrcnn_class_loss: 0.1449 - mrcnn_bbox_loss: 0.4407 - mrcnn_mask_loss: 0.22202025-05-23 17:53:27.440077: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 21s - loss: 1.7650 - rpn_class_loss: 0.0203 - rpn_bbox_loss: 0.8933 - mrcnn_class_loss: 0.1473 - mrcnn_bbox_loss: 0.4645 - mrcnn_mask_loss: 0.23952025-05-23 17:53:27.855883: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:27.860676: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:53:28.241833: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_28 2025-05-23 17:53:28.355625: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_28/marlene.trace.json.gz 2025-05-23 17:53:28.463104: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.059 ms 2025-05-23 17:53:28.535226: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_28Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_28/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_28/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_28/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_28/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 22s - loss: 1.7899 - rpn_class_loss: 0.0181 - rpn_bbox_loss: 0.9037 - mrcnn_class_loss: 0.1406 - mrcnn_bbox_loss: 0.4543 - mrcnn_mask_loss: 0.27312025-05-23 17:53:28.871143: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 20s - loss: 1.8952 - rpn_class_loss: 0.0168 - rpn_bbox_loss: 0.9262 - mrcnn_class_loss: 0.1317 - mrcnn_bbox_loss: 0.5151 - mrcnn_mask_loss: 0.30532025-05-23 17:53:29.174167: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:29.176432: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3439 callback api events and 3439 activity events. 2025-05-23 17:53:29.611855: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_29 2025-05-23 17:53:29.745977: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_29/marlene.trace.json.gz 2025-05-23 17:53:29.846422: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.632 ms 2025-05-23 17:53:29.885732: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_29Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_29/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_29/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_29/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_29/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 20s - loss: 1.8184 - rpn_class_loss: 0.0156 - rpn_bbox_loss: 0.8490 - mrcnn_class_loss: 0.1338 - mrcnn_bbox_loss: 0.5106 - mrcnn_mask_loss: 0.30952025-05-23 17:53:30.196483: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 18s - loss: 1.7708 - rpn_class_loss: 0.0152 - rpn_bbox_loss: 0.7891 - mrcnn_class_loss: 0.1281 - mrcnn_bbox_loss: 0.5318 - mrcnn_mask_loss: 0.30652025-05-23 17:53:30.518452: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:30.519189: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:53:30.679166: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_30 2025-05-23 17:53:30.759614: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_30/marlene.trace.json.gz 2025-05-23 17:53:30.844875: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.735 ms 2025-05-23 17:53:30.889804: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_30/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 18s - loss: 1.7973 - rpn_class_loss: 0.0141 - rpn_bbox_loss: 0.7937 - mrcnn_class_loss: 0.1209 - mrcnn_bbox_loss: 0.5506 - mrcnn_mask_loss: 0.31802025-05-23 17:53:31.180071: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 16s - loss: 1.7397 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.7425 - mrcnn_class_loss: 0.1179 - mrcnn_bbox_loss: 0.5510 - mrcnn_mask_loss: 0.31422025-05-23 17:53:31.449141: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:31.450229: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:53:31.611413: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_31 2025-05-23 17:53:31.709757: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_31/marlene.trace.json.gz 2025-05-23 17:53:31.790817: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.48 ms 2025-05-23 17:53:31.834627: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_31Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_31/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_31/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_31/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_31/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 16s - loss: 1.7847 - rpn_class_loss: 0.0146 - rpn_bbox_loss: 0.7775 - mrcnn_class_loss: 0.1196 - mrcnn_bbox_loss: 0.5618 - mrcnn_mask_loss: 0.31132025-05-23 17:53:32.129600: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 15s - loss: 1.7786 - rpn_class_loss: 0.0162 - rpn_bbox_loss: 0.7780 - mrcnn_class_loss: 0.1140 - mrcnn_bbox_loss: 0.5640 - mrcnn_mask_loss: 0.30642025-05-23 17:53:32.417997: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:32.419481: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3400 callback api events and 3400 activity events. 2025-05-23 17:53:32.680919: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_32 2025-05-23 17:53:32.783432: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_32/marlene.trace.json.gz 2025-05-23 17:53:32.875863: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.578 ms 2025-05-23 17:53:32.922028: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_32Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_32/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_32/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_32/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_32/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 14s - loss: 1.7374 - rpn_class_loss: 0.0159 - rpn_bbox_loss: 0.7500 - mrcnn_class_loss: 0.1132 - mrcnn_bbox_loss: 0.5525 - mrcnn_mask_loss: 0.30572025-05-23 17:53:33.206666: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 17s - loss: 1.7473 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.7884 - mrcnn_class_loss: 0.1115 - mrcnn_bbox_loss: 0.5311 - mrcnn_mask_loss: 0.30032025-05-23 17:53:36.920663: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:36.922344: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3362 callback api events and 3362 activity events. 2025-05-23 17:53:37.172970: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_37 2025-05-23 17:53:37.323485: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_37/marlene.trace.json.gz 2025-05-23 17:53:37.443022: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.048 ms 2025-05-23 17:53:37.499497: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_37Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_37/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_37/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_37/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_37/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 17s - loss: 1.7128 - rpn_class_loss: 0.0173 - rpn_bbox_loss: 0.7820 - mrcnn_class_loss: 0.1074 - mrcnn_bbox_loss: 0.5115 - mrcnn_mask_loss: 0.29472025-05-23 17:53:38.115848: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 23s - loss: 1.6800 - rpn_class_loss: 0.0171 - rpn_bbox_loss: 0.7583 - mrcnn_class_loss: 0.1046 - mrcnn_bbox_loss: 0.5107 - mrcnn_mask_loss: 0.28942025-05-23 17:53:45.956836: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:45.959561: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:53:46.231566: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_46 2025-05-23 17:53:46.328833: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_46/marlene.trace.json.gz 2025-05-23 17:53:46.418956: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.525 ms 2025-05-23 17:53:46.472281: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_46Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_46/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_46/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_46/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_46/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 22s - loss: 1.6891 - rpn_class_loss: 0.0171 - rpn_bbox_loss: 0.7861 - mrcnn_class_loss: 0.1023 - mrcnn_bbox_loss: 0.4985 - mrcnn_mask_loss: 0.28512025-05-23 17:53:46.798445: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 25s - loss: 1.6495 - rpn_class_loss: 0.0166 - rpn_bbox_loss: 0.7605 - mrcnn_class_loss: 0.1026 - mrcnn_bbox_loss: 0.4907 - mrcnn_mask_loss: 0.27912025-05-23 17:53:53.779784: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:53.783233: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3439 callback api events and 3439 activity events. 2025-05-23 17:53:54.074360: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_53 2025-05-23 17:53:54.202420: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_53/marlene.trace.json.gz 2025-05-23 17:53:54.285893: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.017 ms 2025-05-23 17:53:54.332047: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_53Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_53/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_53/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_53/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_53/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 23s - loss: 1.6932 - rpn_class_loss: 0.0177 - rpn_bbox_loss: 0.8202 - mrcnn_class_loss: 0.0997 - mrcnn_bbox_loss: 0.4798 - mrcnn_mask_loss: 0.27582025-05-23 17:53:54.645362: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 22s - loss: 1.6768 - rpn_class_loss: 0.0188 - rpn_bbox_loss: 0.7981 - mrcnn_class_loss: 0.0971 - mrcnn_bbox_loss: 0.4881 - mrcnn_mask_loss: 0.27462025-05-23 17:53:55.847187: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:53:55.848861: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3362 callback api events and 3362 activity events. 2025-05-23 17:53:56.071654: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_55 2025-05-23 17:53:56.184218: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_55/marlene.trace.json.gz 2025-05-23 17:53:56.281529: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.523 ms 2025-05-23 17:53:56.335383: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_55Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_55/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_55/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_55/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_53_55/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 22s - loss: 1.6800 - rpn_class_loss: 0.0189 - rpn_bbox_loss: 0.8131 - mrcnn_class_loss: 0.0945 - mrcnn_bbox_loss: 0.4807 - mrcnn_mask_loss: 0.27272025-05-23 17:54:00.228373: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 22s - loss: 1.6842 - rpn_class_loss: 0.0192 - rpn_bbox_loss: 0.8497 - mrcnn_class_loss: 0.0909 - mrcnn_bbox_loss: 0.4622 - mrcnn_mask_loss: 0.26222025-05-23 17:54:03.567066: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:03.568065: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3255 callback api events and 3255 activity events. 2025-05-23 17:54:03.835520: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_03 2025-05-23 17:54:03.941075: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_03/marlene.trace.json.gz 2025-05-23 17:54:04.020376: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.535 ms 2025-05-23 17:54:04.063849: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_03Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_03/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_03/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_03/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_03/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 21s - loss: 1.6559 - rpn_class_loss: 0.0203 - rpn_bbox_loss: 0.8261 - mrcnn_class_loss: 0.0886 - mrcnn_bbox_loss: 0.4591 - mrcnn_mask_loss: 0.26192025-05-23 17:54:06.936435: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 21s - loss: 1.6306 - rpn_class_loss: 0.0206 - rpn_bbox_loss: 0.8109 - mrcnn_class_loss: 0.0867 - mrcnn_bbox_loss: 0.4501 - mrcnn_mask_loss: 0.26222025-05-23 17:54:12.357811: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:12.359400: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3402 callback api events and 3402 activity events. 2025-05-23 17:54:12.565723: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_12 2025-05-23 17:54:12.670273: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_12/marlene.trace.json.gz 2025-05-23 17:54:12.768666: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.637 ms 2025-05-23 17:54:12.818293: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_12/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 19s - loss: 1.6201 - rpn_class_loss: 0.0208 - rpn_bbox_loss: 0.8191 - mrcnn_class_loss: 0.0839 - mrcnn_bbox_loss: 0.4376 - mrcnn_mask_loss: 0.25882025-05-23 17:54:13.102070: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 19s - loss: 1.6074 - rpn_class_loss: 0.0206 - rpn_bbox_loss: 0.8097 - mrcnn_class_loss: 0.0833 - mrcnn_bbox_loss: 0.4375 - mrcnn_mask_loss: 0.25652025-05-23 17:54:20.714279: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:20.715637: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:54:20.964520: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_20 2025-05-23 17:54:21.067423: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_20/marlene.trace.json.gz 2025-05-23 17:54:21.201887: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.423 ms 2025-05-23 17:54:21.252747: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_20Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_20/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_20/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_20/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_20/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 17s - loss: 1.5773 - rpn_class_loss: 0.0208 - rpn_bbox_loss: 0.7927 - mrcnn_class_loss: 0.0815 - mrcnn_bbox_loss: 0.4278 - mrcnn_mask_loss: 0.25452025-05-23 17:54:21.521981: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 16s - loss: 1.5617 - rpn_class_loss: 0.0206 - rpn_bbox_loss: 0.7829 - mrcnn_class_loss: 0.0811 - mrcnn_bbox_loss: 0.4250 - mrcnn_mask_loss: 0.25202025-05-23 17:54:29.275750: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:29.279356: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3401 activity events. 2025-05-23 17:54:29.605577: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_29 2025-05-23 17:54:29.752147: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_29/marlene.trace.json.gz 2025-05-23 17:54:29.882579: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 2.711 ms 2025-05-23 17:54:29.949790: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_29Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_29/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_29/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_29/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_29/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 14s - loss: 1.5482 - rpn_class_loss: 0.0209 - rpn_bbox_loss: 0.7703 - mrcnn_class_loss: 0.0789 - mrcnn_bbox_loss: 0.4255 - mrcnn_mask_loss: 0.25252025-05-23 17:54:30.327536: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 13s - loss: 1.5553 - rpn_class_loss: 0.0207 - rpn_bbox_loss: 0.7760 - mrcnn_class_loss: 0.0773 - mrcnn_bbox_loss: 0.4291 - mrcnn_mask_loss: 0.25222025-05-23 17:54:38.165420: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:38.167256: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3362 callback api events and 3362 activity events. 2025-05-23 17:54:38.412261: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_38 2025-05-23 17:54:38.511798: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_38/marlene.trace.json.gz 2025-05-23 17:54:38.589405: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.465 ms 2025-05-23 17:54:38.627905: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_38Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_38/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_38/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_38/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_38/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 10s - loss: 1.5364 - rpn_class_loss: 0.0210 - rpn_bbox_loss: 0.7626 - mrcnn_class_loss: 0.0756 - mrcnn_bbox_loss: 0.4246 - mrcnn_mask_loss: 0.25262025-05-23 17:54:38.878825: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 8s - loss: 1.5304 - rpn_class_loss: 0.0208 - rpn_bbox_loss: 0.7629 - mrcnn_class_loss: 0.0745 - mrcnn_bbox_loss: 0.4200 - mrcnn_mask_loss: 0.2522 2025-05-23 17:54:41.726483: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:41.727959: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3438 callback api events and 3438 activity events. 2025-05-23 17:54:41.958816: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_41 2025-05-23 17:54:42.058575: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_41/marlene.trace.json.gz 2025-05-23 17:54:42.142197: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.977 ms 2025-05-23 17:54:42.186272: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_41/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 6s - loss: 1.5296 - rpn_class_loss: 0.0206 - rpn_bbox_loss: 0.7612 - mrcnn_class_loss: 0.0726 - mrcnn_bbox_loss: 0.4243 - mrcnn_mask_loss: 0.25082025-05-23 17:54:45.132191: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 4s - loss: 1.5260 - rpn_class_loss: 0.0204 - rpn_bbox_loss: 0.7593 - mrcnn_class_loss: 0.0710 - mrcnn_bbox_loss: 0.4259 - mrcnn_mask_loss: 0.24942025-05-23 17:54:50.732059: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:50.733031: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3324 callback api events and 3324 activity events. 2025-05-23 17:54:50.937369: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_50 2025-05-23 17:54:51.070697: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_50/marlene.trace.json.gz 2025-05-23 17:54:51.176643: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.915 ms 2025-05-23 17:54:51.234306: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_50Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_50/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_50/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_50/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_54_50/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 2s - loss: 1.5044 - rpn_class_loss: 0.0204 - rpn_bbox_loss: 0.7463 - mrcnn_class_loss: 0.0699 - mrcnn_bbox_loss: 0.4207 - mrcnn_mask_loss: 0.24712025-05-23 17:54:55.009112: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 17:54:59.988744: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 17:54:59.990071: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 3401 callback api events and 3404 activity events. 2025-05-23 17:55:00.266458: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_55_00 2025-05-23 17:55:00.328372: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_55_00/marlene.trace.json.gz 2025-05-23 17:55:00.365406: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 1.023 ms 2025-05-23 17:55:00.386727: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_55_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_55_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_55_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_55_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_17_55_00/marlene.kernel_stats.pb 40/40 [==============================] - 99s 2s/step - loss: 1.4842 - rpn_class_loss: 0.0203 - rpn_bbox_loss: 0.7309 - mrcnn_class_loss: 0.0692 - mrcnn_bbox_loss: 0.4171 - mrcnn_mask_loss: 0.2467 - val_loss: 1.0045 - val_rpn_class_loss: 0.0207 - val_rpn_bbox_loss: 0.3336 - val_mrcnn_class_loss: 0.0237 - val_mrcnn_bbox_loss: 0.4520 - val_mrcnn_mask_loss: 0.1744 2025-05-23 17:55:03.022354: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 17:55:03.071214: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed Training all layers Starting at epoch 2. LR=0.0005 Checkpoint Path: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/mask_rcnn_learn_mask_pancarte_230525_{epoch:04d}.h5 Selecting layers to train conv1 (Conv2D) bn_conv1 (BatchNorm) res2a_branch2a (Conv2D) bn2a_branch2a (BatchNorm) res2a_branch2b (Conv2D) bn2a_branch2b (BatchNorm) res2a_branch2c (Conv2D) res2a_branch1 (Conv2D) bn2a_branch2c (BatchNorm) bn2a_branch1 (BatchNorm) res2b_branch2a (Conv2D) bn2b_branch2a (BatchNorm) res2b_branch2b (Conv2D) bn2b_branch2b (BatchNorm) res2b_branch2c (Conv2D) bn2b_branch2c (BatchNorm) res2c_branch2a (Conv2D) bn2c_branch2a (BatchNorm) res2c_branch2b (Conv2D) bn2c_branch2b (BatchNorm) res2c_branch2c (Conv2D) bn2c_branch2c (BatchNorm) res3a_branch2a (Conv2D) bn3a_branch2a (BatchNorm) res3a_branch2b (Conv2D) bn3a_branch2b (BatchNorm) res3a_branch2c (Conv2D) res3a_branch1 (Conv2D) bn3a_branch2c (BatchNorm) bn3a_branch1 (BatchNorm) res3b_branch2a (Conv2D) bn3b_branch2a (BatchNorm) res3b_branch2b (Conv2D) bn3b_branch2b (BatchNorm) res3b_branch2c (Conv2D) bn3b_branch2c (BatchNorm) res3c_branch2a (Conv2D) bn3c_branch2a (BatchNorm) res3c_branch2b (Conv2D) bn3c_branch2b (BatchNorm) res3c_branch2c (Conv2D) bn3c_branch2c (BatchNorm) res3d_branch2a (Conv2D) bn3d_branch2a (BatchNorm) res3d_branch2b (Conv2D) bn3d_branch2b (BatchNorm) res3d_branch2c (Conv2D) bn3d_branch2c (BatchNorm) res4a_branch2a (Conv2D) bn4a_branch2a (BatchNorm) res4a_branch2b (Conv2D) bn4a_branch2b (BatchNorm) res4a_branch2c (Conv2D) res4a_branch1 (Conv2D) bn4a_branch2c (BatchNorm) bn4a_branch1 (BatchNorm) res4b_branch2a (Conv2D) bn4b_branch2a (BatchNorm) res4b_branch2b (Conv2D) bn4b_branch2b (BatchNorm) res4b_branch2c (Conv2D) bn4b_branch2c (BatchNorm) res4c_branch2a (Conv2D) bn4c_branch2a (BatchNorm) res4c_branch2b (Conv2D) bn4c_branch2b (BatchNorm) res4c_branch2c (Conv2D) bn4c_branch2c (BatchNorm) res4d_branch2a (Conv2D) bn4d_branch2a (BatchNorm) res4d_branch2b (Conv2D) bn4d_branch2b (BatchNorm) res4d_branch2c (Conv2D) bn4d_branch2c (BatchNorm) res4e_branch2a (Conv2D) bn4e_branch2a (BatchNorm) res4e_branch2b (Conv2D) bn4e_branch2b (BatchNorm) res4e_branch2c (Conv2D) bn4e_branch2c (BatchNorm) res4f_branch2a (Conv2D) bn4f_branch2a (BatchNorm) res4f_branch2b (Conv2D) bn4f_branch2b (BatchNorm) res4f_branch2c (Conv2D) bn4f_branch2c (BatchNorm) res4g_branch2a (Conv2D) bn4g_branch2a (BatchNorm) res4g_branch2b (Conv2D) bn4g_branch2b (BatchNorm) res4g_branch2c (Conv2D) bn4g_branch2c (BatchNorm) res4h_branch2a (Conv2D) bn4h_branch2a (BatchNorm) res4h_branch2b (Conv2D) bn4h_branch2b (BatchNorm) res4h_branch2c (Conv2D) bn4h_branch2c (BatchNorm) res4i_branch2a (Conv2D) bn4i_branch2a (BatchNorm) res4i_branch2b (Conv2D) bn4i_branch2b (BatchNorm) res4i_branch2c (Conv2D) bn4i_branch2c (BatchNorm) res4j_branch2a (Conv2D) bn4j_branch2a (BatchNorm) res4j_branch2b (Conv2D) bn4j_branch2b (BatchNorm) res4j_branch2c (Conv2D) bn4j_branch2c (BatchNorm) res4k_branch2a (Conv2D) bn4k_branch2a (BatchNorm) res4k_branch2b (Conv2D) bn4k_branch2b (BatchNorm) res4k_branch2c (Conv2D) bn4k_branch2c (BatchNorm) res4l_branch2a (Conv2D) bn4l_branch2a (BatchNorm) res4l_branch2b (Conv2D) bn4l_branch2b (BatchNorm) res4l_branch2c (Conv2D) bn4l_branch2c (BatchNorm) res4m_branch2a (Conv2D) bn4m_branch2a (BatchNorm) res4m_branch2b (Conv2D) bn4m_branch2b (BatchNorm) res4m_branch2c (Conv2D) bn4m_branch2c (BatchNorm) res4n_branch2a (Conv2D) bn4n_branch2a (BatchNorm) res4n_branch2b (Conv2D) bn4n_branch2b (BatchNorm) res4n_branch2c (Conv2D) bn4n_branch2c (BatchNorm) res4o_branch2a (Conv2D) bn4o_branch2a (BatchNorm) res4o_branch2b (Conv2D) bn4o_branch2b (BatchNorm) res4o_branch2c (Conv2D) bn4o_branch2c (BatchNorm) res4p_branch2a (Conv2D) bn4p_branch2a (BatchNorm) res4p_branch2b (Conv2D) bn4p_branch2b (BatchNorm) res4p_branch2c (Conv2D) bn4p_branch2c (BatchNorm) res4q_branch2a (Conv2D) bn4q_branch2a (BatchNorm) res4q_branch2b (Conv2D) bn4q_branch2b (BatchNorm) res4q_branch2c (Conv2D) bn4q_branch2c (BatchNorm) res4r_branch2a (Conv2D) bn4r_branch2a (BatchNorm) res4r_branch2b (Conv2D) bn4r_branch2b (BatchNorm) res4r_branch2c (Conv2D) bn4r_branch2c (BatchNorm) res4s_branch2a (Conv2D) bn4s_branch2a (BatchNorm) res4s_branch2b (Conv2D) bn4s_branch2b (BatchNorm) res4s_branch2c (Conv2D) bn4s_branch2c (BatchNorm) res4t_branch2a (Conv2D) bn4t_branch2a (BatchNorm) res4t_branch2b (Conv2D) bn4t_branch2b (BatchNorm) res4t_branch2c (Conv2D) bn4t_branch2c (BatchNorm) res4u_branch2a (Conv2D) bn4u_branch2a (BatchNorm) res4u_branch2b (Conv2D) bn4u_branch2b (BatchNorm) res4u_branch2c (Conv2D) bn4u_branch2c (BatchNorm) res4v_branch2a (Conv2D) bn4v_branch2a (BatchNorm) res4v_branch2b (Conv2D) bn4v_branch2b (BatchNorm) res4v_branch2c (Conv2D) bn4v_branch2c (BatchNorm) res4w_branch2a (Conv2D) bn4w_branch2a (BatchNorm) res4w_branch2b (Conv2D) bn4w_branch2b (BatchNorm) res4w_branch2c (Conv2D) bn4w_branch2c (BatchNorm) res5a_branch2a (Conv2D) bn5a_branch2a (BatchNorm) res5a_branch2b (Conv2D) bn5a_branch2b (BatchNorm) res5a_branch2c (Conv2D) res5a_branch1 (Conv2D) bn5a_branch2c (BatchNorm) bn5a_branch1 (BatchNorm) res5b_branch2a (Conv2D) bn5b_branch2a (BatchNorm) res5b_branch2b (Conv2D) bn5b_branch2b (BatchNorm) res5b_branch2c (Conv2D) bn5b_branch2c (BatchNorm) res5c_branch2a (Conv2D) bn5c_branch2a (BatchNorm) res5c_branch2b (Conv2D) bn5c_branch2b (BatchNorm) res5c_branch2c (Conv2D) bn5c_branch2c (BatchNorm) fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.700639). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.464104). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.411106). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.358108). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.381827). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.381336). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.222374). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.403124). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.363328). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.218689). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.290270). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.281097). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.268451). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.276106). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.279702). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.241789). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/engine/training_generator.py:48: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class. warnings.warn( WARNING:tensorflow:Model failed to serialize as JSON. Ignoring... cannot pickle '_thread.RLock' object mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) Epoch 3/6 1/40 [..............................] - ETA: 2:09:08 - loss: 1.0903 - rpn_class_loss: 0.0018 - rpn_bbox_loss: 0.5389 - mrcnn_class_loss: 0.0465 - mrcnn_bbox_loss: 0.3990 - mrcnn_mask_loss: 0.10412025-05-23 18:01:34.214432: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 1:03:27 - loss: 0.8565 - rpn_class_loss: 0.0040 - rpn_bbox_loss: 0.3471 - mrcnn_class_loss: 0.0358 - mrcnn_bbox_loss: 0.3614 - mrcnn_mask_loss: 0.10812025-05-23 18:01:36.251828: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:01:36.288456: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 9995 activity events. 2025-05-23 18:01:37.893748: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_37 2025-05-23 18:01:38.345548: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_37/marlene.trace.json.gz 2025-05-23 18:01:39.129442: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 13.492 ms 2025-05-23 18:01:39.662091: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_37Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_37/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_37/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_37/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_37/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 42:11 - loss: 0.7584 - rpn_class_loss: 0.0061 - rpn_bbox_loss: 0.2732 - mrcnn_class_loss: 0.0448 - mrcnn_bbox_loss: 0.3260 - mrcnn_mask_loss: 0.1083 2025-05-23 18:01:40.787190: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 30:58 - loss: 0.7866 - rpn_class_loss: 0.0049 - rpn_bbox_loss: 0.2268 - mrcnn_class_loss: 0.0782 - mrcnn_bbox_loss: 0.3449 - mrcnn_mask_loss: 0.13182025-05-23 18:01:42.112410: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:01:42.132848: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9957 activity events. 2025-05-23 18:01:44.162122: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_43 2025-05-23 18:01:44.591521: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_43/marlene.trace.json.gz 2025-05-23 18:01:44.953920: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.656 ms 2025-05-23 18:01:45.124296: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_43Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_43/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_43/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_43/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_43/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 24:33 - loss: 0.7128 - rpn_class_loss: 0.0063 - rpn_bbox_loss: 0.2043 - mrcnn_class_loss: 0.0720 - mrcnn_bbox_loss: 0.3015 - mrcnn_mask_loss: 0.12872025-05-23 18:01:45.942474: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 19:59 - loss: 0.7789 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.2121 - mrcnn_class_loss: 0.0678 - mrcnn_bbox_loss: 0.3493 - mrcnn_mask_loss: 0.14212025-05-23 18:01:47.156193: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:01:47.168676: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9957 activity events. 2025-05-23 18:01:49.085019: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_48 2025-05-23 18:01:49.552426: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_48/marlene.trace.json.gz 2025-05-23 18:01:50.192931: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.361 ms 2025-05-23 18:01:50.524105: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_48Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_48/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_48/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_48/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_48/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 16:59 - loss: 0.7637 - rpn_class_loss: 0.0066 - rpn_bbox_loss: 0.1991 - mrcnn_class_loss: 0.0733 - mrcnn_bbox_loss: 0.3342 - mrcnn_mask_loss: 0.15042025-05-23 18:01:51.652407: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 14:27 - loss: 0.8164 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.2489 - mrcnn_class_loss: 0.0672 - mrcnn_bbox_loss: 0.3362 - mrcnn_mask_loss: 0.15552025-05-23 18:01:52.544997: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:01:52.561273: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9918 activity events. 2025-05-23 18:01:54.092229: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_53 2025-05-23 18:01:54.558482: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_53/marlene.trace.json.gz 2025-05-23 18:01:55.213222: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 15.917 ms 2025-05-23 18:01:55.549858: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_53Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_53/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_53/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_53/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_53/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 12:42 - loss: 0.8093 - rpn_class_loss: 0.0101 - rpn_bbox_loss: 0.2404 - mrcnn_class_loss: 0.0663 - mrcnn_bbox_loss: 0.3305 - mrcnn_mask_loss: 0.16202025-05-23 18:01:56.862371: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 11:07 - loss: 0.8188 - rpn_class_loss: 0.0120 - rpn_bbox_loss: 0.2616 - mrcnn_class_loss: 0.0614 - mrcnn_bbox_loss: 0.3198 - mrcnn_mask_loss: 0.16402025-05-23 18:01:57.915541: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:01:57.920879: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9916 callback api events and 9917 activity events. 2025-05-23 18:01:59.834008: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_58 2025-05-23 18:02:00.412652: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_58/marlene.trace.json.gz 2025-05-23 18:02:00.951431: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.632 ms 2025-05-23 18:02:01.271040: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_58Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_58/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_58/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_58/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_01_58/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 9:57 - loss: 0.8820 - rpn_class_loss: 0.0112 - rpn_bbox_loss: 0.2694 - mrcnn_class_loss: 0.0661 - mrcnn_bbox_loss: 0.3480 - mrcnn_mask_loss: 0.1874 2025-05-23 18:02:02.109765: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 8:50 - loss: 0.9297 - rpn_class_loss: 0.0104 - rpn_bbox_loss: 0.2683 - mrcnn_class_loss: 0.0782 - mrcnn_bbox_loss: 0.3661 - mrcnn_mask_loss: 0.20672025-05-23 18:02:02.996801: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:03.000265: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 9995 activity events. 2025-05-23 18:02:04.495631: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_03 2025-05-23 18:02:04.991408: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_03/marlene.trace.json.gz 2025-05-23 18:02:05.271921: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.352 ms 2025-05-23 18:02:05.522046: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_03Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_03/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_03/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_03/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_03/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 7:58 - loss: 1.1864 - rpn_class_loss: 0.0097 - rpn_bbox_loss: 0.4387 - mrcnn_class_loss: 0.0782 - mrcnn_bbox_loss: 0.4394 - mrcnn_mask_loss: 0.22042025-05-23 18:02:06.074234: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 7:10 - loss: 1.2988 - rpn_class_loss: 0.0091 - rpn_bbox_loss: 0.4852 - mrcnn_class_loss: 0.0771 - mrcnn_bbox_loss: 0.4945 - mrcnn_mask_loss: 0.23292025-05-23 18:02:07.228479: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:07.236646: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 9994 activity events. 2025-05-23 18:02:11.262588: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_10 2025-05-23 18:02:11.830884: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_10/marlene.trace.json.gz 2025-05-23 18:02:12.654099: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.587 ms 2025-05-23 18:02:13.232178: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_10Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_10/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_10/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_10/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_10/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 6:38 - loss: 1.2748 - rpn_class_loss: 0.0097 - rpn_bbox_loss: 0.4901 - mrcnn_class_loss: 0.0726 - mrcnn_bbox_loss: 0.4762 - mrcnn_mask_loss: 0.22622025-05-23 18:02:14.605052: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 6:00 - loss: 1.2473 - rpn_class_loss: 0.0103 - rpn_bbox_loss: 0.4932 - mrcnn_class_loss: 0.0681 - mrcnn_bbox_loss: 0.4555 - mrcnn_mask_loss: 0.22022025-05-23 18:02:16.029584: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:16.037441: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9917 activity events. 2025-05-23 18:02:19.492638: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_17 2025-05-23 18:02:20.882869: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_17/marlene.trace.json.gz 2025-05-23 18:02:22.017409: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.574 ms 2025-05-23 18:02:22.435764: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_17Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_17/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_17/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_17/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_17/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 5:35 - loss: 1.2136 - rpn_class_loss: 0.0108 - rpn_bbox_loss: 0.4698 - mrcnn_class_loss: 0.0656 - mrcnn_bbox_loss: 0.4470 - mrcnn_mask_loss: 0.22032025-05-23 18:02:23.445175: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 5:05 - loss: 1.1772 - rpn_class_loss: 0.0112 - rpn_bbox_loss: 0.4484 - mrcnn_class_loss: 0.0632 - mrcnn_bbox_loss: 0.4342 - mrcnn_mask_loss: 0.22022025-05-23 18:02:25.231318: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:25.238304: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9920 activity events. 2025-05-23 18:02:27.347621: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_26 2025-05-23 18:02:27.851014: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_26/marlene.trace.json.gz 2025-05-23 18:02:28.398143: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.069 ms 2025-05-23 18:02:28.677182: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_26/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 4:40 - loss: 1.3205 - rpn_class_loss: 0.0119 - rpn_bbox_loss: 0.5210 - mrcnn_class_loss: 0.0754 - mrcnn_bbox_loss: 0.4669 - mrcnn_mask_loss: 0.24532025-05-23 18:02:29.700251: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 4:15 - loss: 1.4290 - rpn_class_loss: 0.0119 - rpn_bbox_loss: 0.5613 - mrcnn_class_loss: 0.0828 - mrcnn_bbox_loss: 0.5054 - mrcnn_mask_loss: 0.26762025-05-23 18:02:32.425731: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:32.463361: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9997 callback api events and 9999 activity events. 2025-05-23 18:02:36.426714: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_35 2025-05-23 18:02:37.195744: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_35/marlene.trace.json.gz 2025-05-23 18:02:38.540111: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 14.194 ms 2025-05-23 18:02:39.332350: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_35Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_35/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_35/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_35/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_35/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 3:59 - loss: 1.4142 - rpn_class_loss: 0.0126 - rpn_bbox_loss: 0.5721 - mrcnn_class_loss: 0.0801 - mrcnn_bbox_loss: 0.4877 - mrcnn_mask_loss: 0.26182025-05-23 18:02:40.437999: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 3:37 - loss: 1.3946 - rpn_class_loss: 0.0140 - rpn_bbox_loss: 0.5779 - mrcnn_class_loss: 0.0769 - mrcnn_bbox_loss: 0.4711 - mrcnn_mask_loss: 0.25472025-05-23 18:02:41.646421: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:41.663107: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9917 activity events. 2025-05-23 18:02:43.837178: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_42 2025-05-23 18:02:44.414729: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_42/marlene.trace.json.gz 2025-05-23 18:02:45.085230: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.47 ms 2025-05-23 18:02:45.351949: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_42Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_42/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_42/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_42/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_42/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 3:20 - loss: 1.3772 - rpn_class_loss: 0.0139 - rpn_bbox_loss: 0.5774 - mrcnn_class_loss: 0.0748 - mrcnn_bbox_loss: 0.4600 - mrcnn_mask_loss: 0.25112025-05-23 18:02:46.124779: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 3:00 - loss: 1.3525 - rpn_class_loss: 0.0139 - rpn_bbox_loss: 0.5695 - mrcnn_class_loss: 0.0737 - mrcnn_bbox_loss: 0.4485 - mrcnn_mask_loss: 0.2469/usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 2025-05-23 18:02:46.959634: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:46.969126: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9993 callback api events and 9993 activity events. 2025-05-23 18:02:48.091803: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_47 2025-05-23 18:02:48.455383: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_47/marlene.trace.json.gz 2025-05-23 18:02:48.784446: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.703 ms 2025-05-23 18:02:48.908294: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_47Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_47/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_47/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_47/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_47/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 2:44 - loss: 1.3307 - rpn_class_loss: 0.0143 - rpn_bbox_loss: 0.5538 - mrcnn_class_loss: 0.0736 - mrcnn_bbox_loss: 0.4468 - mrcnn_mask_loss: 0.24212025-05-23 18:02:49.515147: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 2:28 - loss: 1.3057 - rpn_class_loss: 0.0147 - rpn_bbox_loss: 0.5386 - mrcnn_class_loss: 0.0728 - mrcnn_bbox_loss: 0.4421 - mrcnn_mask_loss: 0.23742025-05-23 18:02:50.752600: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:50.760042: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9956 activity events. 2025-05-23 18:02:52.745716: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_51 2025-05-23 18:02:53.175674: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_51/marlene.trace.json.gz 2025-05-23 18:02:53.636401: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.893 ms 2025-05-23 18:02:54.063118: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_51Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_51/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_51/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_51/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_51/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 2:14 - loss: 1.3125 - rpn_class_loss: 0.0150 - rpn_bbox_loss: 0.5361 - mrcnn_class_loss: 0.0761 - mrcnn_bbox_loss: 0.4474 - mrcnn_mask_loss: 0.23792025-05-23 18:02:55.073441: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 2:00 - loss: 1.3104 - rpn_class_loss: 0.0151 - rpn_bbox_loss: 0.5313 - mrcnn_class_loss: 0.0794 - mrcnn_bbox_loss: 0.4471 - mrcnn_mask_loss: 0.23762025-05-23 18:02:56.726335: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:02:56.748779: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9997 callback api events and 10000 activity events. 2025-05-23 18:02:59.360393: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_58 2025-05-23 18:02:59.688767: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_58/marlene.trace.json.gz 2025-05-23 18:03:00.305346: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 16.533 ms 2025-05-23 18:03:00.853859: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_58Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_58/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_58/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_58/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_02_58/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 1:48 - loss: 1.2983 - rpn_class_loss: 0.0158 - rpn_bbox_loss: 0.5280 - mrcnn_class_loss: 0.0774 - mrcnn_bbox_loss: 0.4410 - mrcnn_mask_loss: 0.23612025-05-23 18:03:02.002592: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 1:35 - loss: 1.2797 - rpn_class_loss: 0.0162 - rpn_bbox_loss: 0.5226 - mrcnn_class_loss: 0.0756 - mrcnn_bbox_loss: 0.4312 - mrcnn_mask_loss: 0.23412025-05-23 18:03:03.784195: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:03:03.815598: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9916 callback api events and 9916 activity events. 2025-05-23 18:03:06.578260: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_05 2025-05-23 18:03:07.453707: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_05/marlene.trace.json.gz 2025-05-23 18:03:08.686559: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 17.812 ms 2025-05-23 18:03:09.425180: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_05Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_05/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_05/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_05/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_05/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 1:25 - loss: 1.2636 - rpn_class_loss: 0.0159 - rpn_bbox_loss: 0.5203 - mrcnn_class_loss: 0.0741 - mrcnn_bbox_loss: 0.4214 - mrcnn_mask_loss: 0.23192025-05-23 18:03:10.387980: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 1:13 - loss: 1.2434 - rpn_class_loss: 0.0155 - rpn_bbox_loss: 0.5153 - mrcnn_class_loss: 0.0722 - mrcnn_bbox_loss: 0.4113 - mrcnn_mask_loss: 0.22912025-05-23 18:03:11.503301: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:03:11.521430: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9916 callback api events and 9919 activity events. 2025-05-23 18:03:13.437027: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_12 2025-05-23 18:03:14.087815: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_12/marlene.trace.json.gz 2025-05-23 18:03:15.054050: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 15.106 ms 2025-05-23 18:03:15.342110: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_12/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 1:03 - loss: 1.2357 - rpn_class_loss: 0.0156 - rpn_bbox_loss: 0.5152 - mrcnn_class_loss: 0.0713 - mrcnn_bbox_loss: 0.4080 - mrcnn_mask_loss: 0.22562025-05-23 18:03:16.638817: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 53s - loss: 1.2144 - rpn_class_loss: 0.0157 - rpn_bbox_loss: 0.5023 - mrcnn_class_loss: 0.0694 - mrcnn_bbox_loss: 0.4044 - mrcnn_mask_loss: 0.2226 2025-05-23 18:03:18.262877: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:03:18.279008: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9879 callback api events and 9879 activity events. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 2025-05-23 18:03:19.874919: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_19 2025-05-23 18:03:20.169685: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_19/marlene.trace.json.gz 2025-05-23 18:03:20.452951: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.133 ms 2025-05-23 18:03:20.634239: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_19Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_19/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_19/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_19/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_19/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 43s - loss: 1.1991 - rpn_class_loss: 0.0157 - rpn_bbox_loss: 0.4978 - mrcnn_class_loss: 0.0685 - mrcnn_bbox_loss: 0.3974 - mrcnn_mask_loss: 0.21972025-05-23 18:03:21.452457: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 34s - loss: 1.1956 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.5000 - mrcnn_class_loss: 0.0668 - mrcnn_bbox_loss: 0.3948 - mrcnn_mask_loss: 0.21792025-05-23 18:03:23.774262: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:03:23.793709: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9920 activity events. 2025-05-23 18:03:25.830765: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_25 2025-05-23 18:03:26.554165: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_25/marlene.trace.json.gz 2025-05-23 18:03:27.309347: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 23.94 ms 2025-05-23 18:03:28.284041: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_25Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_25/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_25/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_25/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_25/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 25s - loss: 1.1657 - rpn_class_loss: 0.0161 - rpn_bbox_loss: 0.4884 - mrcnn_class_loss: 0.0650 - mrcnn_bbox_loss: 0.3841 - mrcnn_mask_loss: 0.21202025-05-23 18:03:29.415344: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 16s - loss: 1.1844 - rpn_class_loss: 0.0163 - rpn_bbox_loss: 0.4990 - mrcnn_class_loss: 0.0647 - mrcnn_bbox_loss: 0.3914 - mrcnn_mask_loss: 0.21292025-05-23 18:03:30.825449: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:03:30.832780: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 9997 activity events. 2025-05-23 18:03:34.216324: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_32 2025-05-23 18:03:34.816279: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_32/marlene.trace.json.gz 2025-05-23 18:03:35.880330: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 17.003 ms 2025-05-23 18:03:36.603382: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_32Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_32/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_32/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_32/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_32/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 8s - loss: 1.1798 - rpn_class_loss: 0.0162 - rpn_bbox_loss: 0.4994 - mrcnn_class_loss: 0.0637 - mrcnn_bbox_loss: 0.3900 - mrcnn_mask_loss: 0.2105 2025-05-23 18:03:37.713035: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 18:03:39.355234: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:03:39.360631: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9921 activity events. 2025-05-23 18:03:41.278833: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_40 2025-05-23 18:03:41.696643: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_40/marlene.trace.json.gz 2025-05-23 18:03:42.053611: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.522 ms 2025-05-23 18:03:42.247856: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_40Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_40/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_40/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_40/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_03_40/marlene.kernel_stats.pb 40/40 [==============================] - 359s 9s/step - loss: 1.1772 - rpn_class_loss: 0.0166 - rpn_bbox_loss: 0.4993 - mrcnn_class_loss: 0.0630 - mrcnn_bbox_loss: 0.3880 - mrcnn_mask_loss: 0.2103 - val_loss: 1.0531 - val_rpn_class_loss: 0.0170 - val_rpn_bbox_loss: 0.5952 - val_mrcnn_class_loss: 0.0267 - val_mrcnn_bbox_loss: 0.2655 - val_mrcnn_mask_loss: 0.1486 Epoch 4/6 1/40 [..............................] - ETA: 28s - loss: 2.6909 - rpn_class_loss: 0.0396 - rpn_bbox_loss: 1.8322 - mrcnn_class_loss: 0.0675 - mrcnn_bbox_loss: 0.5240 - mrcnn_mask_loss: 0.22762025-05-23 18:04:19.343133: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 32s - loss: 1.5879 - rpn_class_loss: 0.0252 - rpn_bbox_loss: 0.9788 - mrcnn_class_loss: 0.0520 - mrcnn_bbox_loss: 0.3434 - mrcnn_mask_loss: 0.18852025-05-23 18:04:20.569499: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:04:20.585765: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9960 activity events. 2025-05-23 18:04:21.570099: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_21 2025-05-23 18:04:21.808928: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_21/marlene.trace.json.gz 2025-05-23 18:04:22.094566: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.377 ms 2025-05-23 18:04:22.269677: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_21Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_21/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_21/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_21/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_21/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 52s - loss: 1.3190 - rpn_class_loss: 0.0201 - rpn_bbox_loss: 0.8172 - mrcnn_class_loss: 0.0465 - mrcnn_bbox_loss: 0.2654 - mrcnn_mask_loss: 0.16982025-05-23 18:04:22.874885: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 44s - loss: 1.2904 - rpn_class_loss: 0.0184 - rpn_bbox_loss: 0.7697 - mrcnn_class_loss: 0.0438 - mrcnn_bbox_loss: 0.2960 - mrcnn_mask_loss: 0.16242025-05-23 18:04:23.786118: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:04:23.797137: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9921 activity events. 2025-05-23 18:04:27.291380: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_25 2025-05-23 18:04:27.767673: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_25/marlene.trace.json.gz 2025-05-23 18:04:28.826293: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 13.483 ms 2025-05-23 18:04:29.233767: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_25Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_25/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_25/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_25/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_25/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 1:26 - loss: 1.1242 - rpn_class_loss: 0.0171 - rpn_bbox_loss: 0.6407 - mrcnn_class_loss: 0.0409 - mrcnn_bbox_loss: 0.2639 - mrcnn_mask_loss: 0.16172025-05-23 18:04:30.929572: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 1:19 - loss: 1.1096 - rpn_class_loss: 0.0159 - rpn_bbox_loss: 0.6349 - mrcnn_class_loss: 0.0386 - mrcnn_bbox_loss: 0.2635 - mrcnn_mask_loss: 0.15672025-05-23 18:04:32.948450: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:04:32.964806: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9918 activity events. 2025-05-23 18:04:34.345148: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_33 2025-05-23 18:04:34.874077: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_33/marlene.trace.json.gz 2025-05-23 18:04:35.764665: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 17.178 ms 2025-05-23 18:04:36.461359: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_33Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_33/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_33/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_33/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_33/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 1:28 - loss: 1.1168 - rpn_class_loss: 0.0155 - rpn_bbox_loss: 0.5663 - mrcnn_class_loss: 0.0506 - mrcnn_bbox_loss: 0.3104 - mrcnn_mask_loss: 0.17402025-05-23 18:04:37.350441: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 1:19 - loss: 1.1102 - rpn_class_loss: 0.0152 - rpn_bbox_loss: 0.5157 - mrcnn_class_loss: 0.0561 - mrcnn_bbox_loss: 0.3308 - mrcnn_mask_loss: 0.19242025-05-23 18:04:39.016821: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:04:39.040894: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 9994 activity events. 2025-05-23 18:04:40.349148: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_39 2025-05-23 18:04:40.960804: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_39/marlene.trace.json.gz 2025-05-23 18:04:41.945353: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 17.708 ms 2025-05-23 18:04:42.572463: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_39Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_39/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_39/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_39/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_39/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 1:26 - loss: 1.0399 - rpn_class_loss: 0.0171 - rpn_bbox_loss: 0.4747 - mrcnn_class_loss: 0.0518 - mrcnn_bbox_loss: 0.3146 - mrcnn_mask_loss: 0.18152025-05-23 18:04:43.698330: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 1:30 - loss: 0.9804 - rpn_class_loss: 0.0192 - rpn_bbox_loss: 0.4389 - mrcnn_class_loss: 0.0491 - mrcnn_bbox_loss: 0.3003 - mrcnn_mask_loss: 0.17302025-05-23 18:04:49.233497: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:04:49.247652: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 9967 activity events. 2025-05-23 18:04:51.846000: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_51 2025-05-23 18:04:52.351439: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_51/marlene.trace.json.gz 2025-05-23 18:04:52.770157: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 10.914 ms 2025-05-23 18:04:53.144952: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_51Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_51/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_51/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_51/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_04_51/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 1:33 - loss: 0.9573 - rpn_class_loss: 0.0180 - rpn_bbox_loss: 0.4196 - mrcnn_class_loss: 0.0475 - mrcnn_bbox_loss: 0.3023 - mrcnn_mask_loss: 0.16992025-05-23 18:04:54.096855: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 1:55 - loss: 0.9350 - rpn_class_loss: 0.0178 - rpn_bbox_loss: 0.4105 - mrcnn_class_loss: 0.0446 - mrcnn_bbox_loss: 0.2965 - mrcnn_mask_loss: 0.16562025-05-23 18:05:08.672560: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:05:08.692197: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9958 callback api events and 9990 activity events. 2025-05-23 18:05:10.686820: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_09 2025-05-23 18:05:11.403284: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_09/marlene.trace.json.gz 2025-05-23 18:05:12.179082: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 13.155 ms 2025-05-23 18:05:12.439442: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_09/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 1:54 - loss: 0.9129 - rpn_class_loss: 0.0173 - rpn_bbox_loss: 0.3913 - mrcnn_class_loss: 0.0434 - mrcnn_bbox_loss: 0.2976 - mrcnn_mask_loss: 0.16332025-05-23 18:05:13.612159: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 1:53 - loss: 0.8849 - rpn_class_loss: 0.0171 - rpn_bbox_loss: 0.3827 - mrcnn_class_loss: 0.0411 - mrcnn_bbox_loss: 0.2839 - mrcnn_mask_loss: 0.16012025-05-23 18:05:19.888894: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:05:19.912109: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9920 callback api events and 9935 activity events. 2025-05-23 18:05:21.098659: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_20 2025-05-23 18:05:21.378090: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_20/marlene.trace.json.gz 2025-05-23 18:05:21.713365: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.456 ms 2025-05-23 18:05:21.956672: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_20Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_20/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_20/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_20/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_20/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 1:58 - loss: 0.8482 - rpn_class_loss: 0.0165 - rpn_bbox_loss: 0.3583 - mrcnn_class_loss: 0.0399 - mrcnn_bbox_loss: 0.2752 - mrcnn_mask_loss: 0.15842025-05-23 18:05:29.655529: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 1:55 - loss: 0.8142 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.3395 - mrcnn_class_loss: 0.0388 - mrcnn_bbox_loss: 0.2628 - mrcnn_mask_loss: 0.15702025-05-23 18:05:36.496102: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:05:36.525916: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9976 activity events. 2025-05-23 18:05:38.168240: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_37 2025-05-23 18:05:38.544126: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_37/marlene.trace.json.gz 2025-05-23 18:05:39.094389: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.347 ms 2025-05-23 18:05:39.314360: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_37Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_37/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_37/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_37/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_37/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 1:57 - loss: 0.7889 - rpn_class_loss: 0.0160 - rpn_bbox_loss: 0.3220 - mrcnn_class_loss: 0.0417 - mrcnn_bbox_loss: 0.2565 - mrcnn_mask_loss: 0.15282025-05-23 18:05:45.316229: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 1:58 - loss: 0.7615 - rpn_class_loss: 0.0159 - rpn_bbox_loss: 0.3062 - mrcnn_class_loss: 0.0413 - mrcnn_bbox_loss: 0.2494 - mrcnn_mask_loss: 0.14872025-05-23 18:05:55.704167: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:05:55.727968: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10019 activity events. 2025-05-23 18:05:57.709281: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_56 2025-05-23 18:05:58.237797: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_56/marlene.trace.json.gz 2025-05-23 18:05:58.787524: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.43 ms 2025-05-23 18:05:59.189307: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_56Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_56/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_56/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_56/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_05_56/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 2:01 - loss: 0.7392 - rpn_class_loss: 0.0153 - rpn_bbox_loss: 0.2925 - mrcnn_class_loss: 0.0406 - mrcnn_bbox_loss: 0.2450 - mrcnn_mask_loss: 0.14582025-05-23 18:06:08.929914: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 1:58 - loss: 0.7205 - rpn_class_loss: 0.0152 - rpn_bbox_loss: 0.2835 - mrcnn_class_loss: 0.0394 - mrcnn_bbox_loss: 0.2394 - mrcnn_mask_loss: 0.14292025-05-23 18:06:17.643828: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:06:17.666273: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9943 activity events. 2025-05-23 18:06:19.181798: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_18 2025-05-23 18:06:19.588560: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_18/marlene.trace.json.gz 2025-05-23 18:06:20.145815: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.493 ms 2025-05-23 18:06:20.393490: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_18Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_18/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_18/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_18/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_18/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 1:53 - loss: 0.7202 - rpn_class_loss: 0.0153 - rpn_bbox_loss: 0.2859 - mrcnn_class_loss: 0.0378 - mrcnn_bbox_loss: 0.2404 - mrcnn_mask_loss: 0.14092025-05-23 18:06:23.868755: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 1:56 - loss: 0.7173 - rpn_class_loss: 0.0154 - rpn_bbox_loss: 0.2877 - mrcnn_class_loss: 0.0363 - mrcnn_bbox_loss: 0.2394 - mrcnn_mask_loss: 0.13852025-05-23 18:06:41.740035: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:06:41.758956: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 9995 activity events. 2025-05-23 18:06:45.057712: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_43 2025-05-23 18:06:45.643050: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_43/marlene.trace.json.gz 2025-05-23 18:06:46.988958: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.975 ms 2025-05-23 18:06:47.256845: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_43Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_43/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_43/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_43/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_06_43/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 1:52 - loss: 0.7271 - rpn_class_loss: 0.0148 - rpn_bbox_loss: 0.2995 - mrcnn_class_loss: 0.0357 - mrcnn_bbox_loss: 0.2390 - mrcnn_mask_loss: 0.13802025-05-23 18:06:51.050906: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 1:51 - loss: 0.7437 - rpn_class_loss: 0.0145 - rpn_bbox_loss: 0.3163 - mrcnn_class_loss: 0.0345 - mrcnn_bbox_loss: 0.2417 - mrcnn_mask_loss: 0.13662025-05-23 18:07:06.915310: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:07:06.927534: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9958 activity events. 2025-05-23 18:07:08.967194: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_08 2025-05-23 18:07:09.323113: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_08/marlene.trace.json.gz 2025-05-23 18:07:09.687844: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 10.365 ms 2025-05-23 18:07:09.982041: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_08Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_08/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_08/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_08/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_08/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 1:43 - loss: 0.7469 - rpn_class_loss: 0.0144 - rpn_bbox_loss: 0.3162 - mrcnn_class_loss: 0.0338 - mrcnn_bbox_loss: 0.2466 - mrcnn_mask_loss: 0.13592025-05-23 18:07:10.815723: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 26/40 [==================>...........] - ETA: 1:44 - loss: 0.7444 - rpn_class_loss: 0.0142 - rpn_bbox_loss: 0.3135 - mrcnn_class_loss: 0.0336 - mrcnn_bbox_loss: 0.2475 - mrcnn_mask_loss: 0.13562025-05-23 18:07:33.171510: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:07:33.195346: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10011 activity events. 2025-05-23 18:07:34.499144: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_33 2025-05-23 18:07:34.981645: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_33/marlene.trace.json.gz 2025-05-23 18:07:35.484299: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.26 ms 2025-05-23 18:07:35.721319: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_33Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_33/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_33/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_33/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_07_33/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 1:39 - loss: 0.7298 - rpn_class_loss: 0.0139 - rpn_bbox_loss: 0.3043 - mrcnn_class_loss: 0.0329 - mrcnn_bbox_loss: 0.2453 - mrcnn_mask_loss: 0.13332025-05-23 18:07:45.485730: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 1:35 - loss: 0.7188 - rpn_class_loss: 0.0137 - rpn_bbox_loss: 0.2965 - mrcnn_class_loss: 0.0326 - mrcnn_bbox_loss: 0.2444 - mrcnn_mask_loss: 0.13162025-05-23 18:08:01.228084: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:08:01.234392: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9966 activity events. 2025-05-23 18:08:02.981907: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_02 2025-05-23 18:08:03.701402: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_02/marlene.trace.json.gz 2025-05-23 18:08:04.292521: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 16.689 ms 2025-05-23 18:08:04.775080: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_02Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_02/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_02/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_02/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_02/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 1:28 - loss: 0.7127 - rpn_class_loss: 0.0133 - rpn_bbox_loss: 0.2975 - mrcnn_class_loss: 0.0317 - mrcnn_bbox_loss: 0.2398 - mrcnn_mask_loss: 0.13042025-05-23 18:08:10.928989: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 1:23 - loss: 0.7055 - rpn_class_loss: 0.0130 - rpn_bbox_loss: 0.2971 - mrcnn_class_loss: 0.0309 - mrcnn_bbox_loss: 0.2354 - mrcnn_mask_loss: 0.12912025-05-23 18:08:29.604768: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:08:29.626479: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10048 activity events. 2025-05-23 18:08:31.271705: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_30 2025-05-23 18:08:31.976376: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_30/marlene.trace.json.gz 2025-05-23 18:08:32.548810: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 10.078 ms 2025-05-23 18:08:32.905161: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_08_30/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 1:14 - loss: 0.7116 - rpn_class_loss: 0.0130 - rpn_bbox_loss: 0.3016 - mrcnn_class_loss: 0.0302 - mrcnn_bbox_loss: 0.2379 - mrcnn_mask_loss: 0.12902025-05-23 18:08:35.013983: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 1:11 - loss: 0.7110 - rpn_class_loss: 0.0130 - rpn_bbox_loss: 0.3008 - mrcnn_class_loss: 0.0295 - mrcnn_bbox_loss: 0.2390 - mrcnn_mask_loss: 0.12862025-05-23 18:09:05.015019: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:09:05.032951: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10046 activity events. 2025-05-23 18:09:07.143152: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_06 2025-05-23 18:09:07.753282: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_06/marlene.trace.json.gz 2025-05-23 18:09:08.426417: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.929 ms 2025-05-23 18:09:08.649009: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_06Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_06/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_06/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_06/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_06/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 1:01 - loss: 0.7263 - rpn_class_loss: 0.0127 - rpn_bbox_loss: 0.2940 - mrcnn_class_loss: 0.0361 - mrcnn_bbox_loss: 0.2486 - mrcnn_mask_loss: 0.13492025-05-23 18:09:09.876598: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 55s - loss: 0.7306 - rpn_class_loss: 0.0124 - rpn_bbox_loss: 0.2984 - mrcnn_class_loss: 0.0355 - mrcnn_bbox_loss: 0.2504 - mrcnn_mask_loss: 0.1338 2025-05-23 18:09:33.707580: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:09:33.719825: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10022 activity events. 2025-05-23 18:09:34.697279: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_34 2025-05-23 18:09:35.031277: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_34/marlene.trace.json.gz 2025-05-23 18:09:35.448191: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.201 ms 2025-05-23 18:09:35.710779: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_34/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 45s - loss: 0.7350 - rpn_class_loss: 0.0122 - rpn_bbox_loss: 0.2903 - mrcnn_class_loss: 0.0404 - mrcnn_bbox_loss: 0.2533 - mrcnn_mask_loss: 0.13872025-05-23 18:09:36.580439: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 37s - loss: 0.7292 - rpn_class_loss: 0.0121 - rpn_bbox_loss: 0.2858 - mrcnn_class_loss: 0.0398 - mrcnn_bbox_loss: 0.2546 - mrcnn_mask_loss: 0.13692025-05-23 18:09:56.859384: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:09:56.892230: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9977 activity events. 2025-05-23 18:09:59.848003: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_58 2025-05-23 18:10:00.626953: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_58/marlene.trace.json.gz 2025-05-23 18:10:01.276622: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.752 ms 2025-05-23 18:10:01.685010: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_58Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_58/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_58/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_58/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_09_58/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 27s - loss: 0.7254 - rpn_class_loss: 0.0120 - rpn_bbox_loss: 0.2869 - mrcnn_class_loss: 0.0391 - mrcnn_bbox_loss: 0.2518 - mrcnn_mask_loss: 0.13582025-05-23 18:10:02.580055: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 38/40 [===========================>..] - ETA: 19s - loss: 0.7177 - rpn_class_loss: 0.0120 - rpn_bbox_loss: 0.2825 - mrcnn_class_loss: 0.0384 - mrcnn_bbox_loss: 0.2504 - mrcnn_mask_loss: 0.13432025-05-23 18:10:22.333653: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:10:22.351341: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 10009 activity events. 2025-05-23 18:10:23.707186: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_23 2025-05-23 18:10:24.059084: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_23/marlene.trace.json.gz 2025-05-23 18:10:24.294641: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.579 ms 2025-05-23 18:10:24.432568: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_23/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 9s - loss: 0.7121 - rpn_class_loss: 0.0120 - rpn_bbox_loss: 0.2818 - mrcnn_class_loss: 0.0380 - mrcnn_bbox_loss: 0.2474 - mrcnn_mask_loss: 0.1329 2025-05-23 18:10:25.236251: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 18:10:51.596542: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:10:51.629113: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9990 activity events. 2025-05-23 18:10:53.448763: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_52 2025-05-23 18:10:54.152604: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_52/marlene.trace.json.gz 2025-05-23 18:10:54.953477: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.885 ms 2025-05-23 18:10:55.272313: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_52Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_52/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_52/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_52/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_10_52/marlene.kernel_stats.pb 40/40 [==============================] - 417s 10s/step - loss: 0.7051 - rpn_class_loss: 0.0118 - rpn_bbox_loss: 0.2769 - mrcnn_class_loss: 0.0377 - mrcnn_bbox_loss: 0.2462 - mrcnn_mask_loss: 0.1326 - val_loss: 0.3879 - val_rpn_class_loss: 0.0091 - val_rpn_bbox_loss: 0.1162 - val_mrcnn_class_loss: 0.0201 - val_mrcnn_bbox_loss: 0.1471 - val_mrcnn_mask_loss: 0.0954 Epoch 5/6 1/40 [..............................] - ETA: 25s - loss: 0.3990 - rpn_class_loss: 0.0136 - rpn_bbox_loss: 0.1143 - mrcnn_class_loss: 0.0203 - mrcnn_bbox_loss: 0.1626 - mrcnn_mask_loss: 0.08842025-05-23 18:11:21.023988: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 1:52 - loss: 1.1111 - rpn_class_loss: 0.0157 - rpn_bbox_loss: 0.8205 - mrcnn_class_loss: 0.0253 - mrcnn_bbox_loss: 0.1743 - mrcnn_mask_loss: 0.07542025-05-23 18:11:27.352954: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:11:27.368057: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9993 callback api events and 10014 activity events. 2025-05-23 18:11:29.870406: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_29 2025-05-23 18:11:30.544866: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_29/marlene.trace.json.gz 2025-05-23 18:11:31.163843: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.768 ms 2025-05-23 18:11:31.585898: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_29Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_29/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_29/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_29/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_29/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 2:26 - loss: 0.8741 - rpn_class_loss: 0.0125 - rpn_bbox_loss: 0.5691 - mrcnn_class_loss: 0.0277 - mrcnn_bbox_loss: 0.1758 - mrcnn_mask_loss: 0.08902025-05-23 18:11:32.272564: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 4/40 [==>...........................] - ETA: 4:28 - loss: 0.8824 - rpn_class_loss: 0.0116 - rpn_bbox_loss: 0.5245 - mrcnn_class_loss: 0.0325 - mrcnn_bbox_loss: 0.1883 - mrcnn_mask_loss: 0.12562025-05-23 18:11:50.769628: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:11:50.784327: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9917 callback api events and 9980 activity events. 2025-05-23 18:11:52.325600: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_51 2025-05-23 18:11:52.887907: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_51/marlene.trace.json.gz 2025-05-23 18:11:53.393934: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 40.624 ms 2025-05-23 18:11:53.614276: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_51Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_51/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_51/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_51/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_11_51/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 3:58 - loss: 1.0042 - rpn_class_loss: 0.0120 - rpn_bbox_loss: 0.6478 - mrcnn_class_loss: 0.0327 - mrcnn_bbox_loss: 0.1992 - mrcnn_mask_loss: 0.11262025-05-23 18:11:54.371201: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 6/40 [===>..........................] - ETA: 4:33 - loss: 0.9565 - rpn_class_loss: 0.0108 - rpn_bbox_loss: 0.5683 - mrcnn_class_loss: 0.0388 - mrcnn_bbox_loss: 0.2188 - mrcnn_mask_loss: 0.11982025-05-23 18:12:08.973809: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:12:08.978829: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9993 callback api events and 10049 activity events. 2025-05-23 18:12:09.746280: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_09 2025-05-23 18:12:10.071359: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_09/marlene.trace.json.gz 2025-05-23 18:12:10.364995: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.465 ms 2025-05-23 18:12:10.515959: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_09/marlene.kernel_stats.pb 7/40 [====>.........................] - ETA: 3:58 - loss: 0.9112 - rpn_class_loss: 0.0107 - rpn_bbox_loss: 0.5178 - mrcnn_class_loss: 0.0384 - mrcnn_bbox_loss: 0.2180 - mrcnn_mask_loss: 0.12642025-05-23 18:12:11.000942: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 8/40 [=====>........................] - ETA: 4:11 - loss: 0.8571 - rpn_class_loss: 0.0099 - rpn_bbox_loss: 0.4664 - mrcnn_class_loss: 0.0363 - mrcnn_bbox_loss: 0.2232 - mrcnn_mask_loss: 0.12142025-05-23 18:12:23.500510: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:12:23.509065: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9998 activity events. 2025-05-23 18:12:24.312192: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_23 2025-05-23 18:12:24.613566: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_23/marlene.trace.json.gz 2025-05-23 18:12:25.021364: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.129 ms 2025-05-23 18:12:25.243585: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_23/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 3:45 - loss: 0.8198 - rpn_class_loss: 0.0090 - rpn_bbox_loss: 0.4227 - mrcnn_class_loss: 0.0406 - mrcnn_bbox_loss: 0.2223 - mrcnn_mask_loss: 0.12522025-05-23 18:12:25.734282: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 3:51 - loss: 0.7672 - rpn_class_loss: 0.0084 - rpn_bbox_loss: 0.3847 - mrcnn_class_loss: 0.0400 - mrcnn_bbox_loss: 0.2129 - mrcnn_mask_loss: 0.12122025-05-23 18:12:37.866313: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:12:37.871592: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9993 callback api events and 10035 activity events. 2025-05-23 18:12:38.727002: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_38 2025-05-23 18:12:39.008987: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_38/marlene.trace.json.gz 2025-05-23 18:12:39.258263: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.911 ms 2025-05-23 18:12:39.463102: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_38Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_38/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_38/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_38/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_38/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 3:32 - loss: 0.8259 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.3611 - mrcnn_class_loss: 0.0440 - mrcnn_bbox_loss: 0.2806 - mrcnn_mask_loss: 0.13162025-05-23 18:12:40.921637: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 3:31 - loss: 0.8526 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.3385 - mrcnn_class_loss: 0.0468 - mrcnn_bbox_loss: 0.3200 - mrcnn_mask_loss: 0.13862025-05-23 18:12:51.294877: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:12:51.303708: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10029 activity events. 2025-05-23 18:12:52.031808: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_51 2025-05-23 18:12:52.350934: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_51/marlene.trace.json.gz 2025-05-23 18:12:52.623876: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.497 ms 2025-05-23 18:12:52.766345: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_51Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_51/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_51/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_51/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_12_51/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 3:13 - loss: 0.8257 - rpn_class_loss: 0.0082 - rpn_bbox_loss: 0.3300 - mrcnn_class_loss: 0.0444 - mrcnn_bbox_loss: 0.3058 - mrcnn_mask_loss: 0.13732025-05-23 18:12:53.428607: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 3:15 - loss: 0.8112 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.3300 - mrcnn_class_loss: 0.0420 - mrcnn_bbox_loss: 0.2962 - mrcnn_mask_loss: 0.13522025-05-23 18:13:06.213023: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:13:06.222173: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 10003 activity events. 2025-05-23 18:13:06.965681: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_06 2025-05-23 18:13:07.427507: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_06/marlene.trace.json.gz 2025-05-23 18:13:07.922770: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.523 ms 2025-05-23 18:13:08.155256: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_06Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_06/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_06/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_06/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_06/marlene.kernel_stats.pb 15/40 [==========>...................] - ETA: 3:05 - loss: 0.8141 - rpn_class_loss: 0.0085 - rpn_bbox_loss: 0.3348 - mrcnn_class_loss: 0.0431 - mrcnn_bbox_loss: 0.2899 - mrcnn_mask_loss: 0.13792025-05-23 18:13:11.831265: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 2:55 - loss: 0.8210 - rpn_class_loss: 0.0089 - rpn_bbox_loss: 0.3400 - mrcnn_class_loss: 0.0453 - mrcnn_bbox_loss: 0.2852 - mrcnn_mask_loss: 0.14162025-05-23 18:13:17.677637: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:13:17.681160: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9980 activity events. 2025-05-23 18:13:18.362880: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_18 2025-05-23 18:13:18.684383: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_18/marlene.trace.json.gz 2025-05-23 18:13:19.107287: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.035 ms 2025-05-23 18:13:19.332661: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_18Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_18/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_18/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_18/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_18/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 2:49 - loss: 0.7862 - rpn_class_loss: 0.0086 - rpn_bbox_loss: 0.3212 - mrcnn_class_loss: 0.0435 - mrcnn_bbox_loss: 0.2730 - mrcnn_mask_loss: 0.13992025-05-23 18:13:25.491148: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 2:39 - loss: 0.7563 - rpn_class_loss: 0.0083 - rpn_bbox_loss: 0.3050 - mrcnn_class_loss: 0.0419 - mrcnn_bbox_loss: 0.2635 - mrcnn_mask_loss: 0.13752025-05-23 18:13:30.893802: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:13:30.899071: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9980 activity events. 2025-05-23 18:13:31.797516: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_31 2025-05-23 18:13:32.073056: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_31/marlene.trace.json.gz 2025-05-23 18:13:32.447264: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.759 ms 2025-05-23 18:13:32.672586: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_31Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_31/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_31/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_31/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_31/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 2:33 - loss: 0.7352 - rpn_class_loss: 0.0082 - rpn_bbox_loss: 0.2942 - mrcnn_class_loss: 0.0406 - mrcnn_bbox_loss: 0.2532 - mrcnn_mask_loss: 0.13902025-05-23 18:13:39.332776: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 2:23 - loss: 0.7160 - rpn_class_loss: 0.0080 - rpn_bbox_loss: 0.2848 - mrcnn_class_loss: 0.0410 - mrcnn_bbox_loss: 0.2424 - mrcnn_mask_loss: 0.13982025-05-23 18:13:44.420312: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:13:44.423849: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 9979 activity events. 2025-05-23 18:13:45.378709: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_45 2025-05-23 18:13:45.675844: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_45/marlene.trace.json.gz 2025-05-23 18:13:46.077660: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.541 ms 2025-05-23 18:13:46.348537: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_45Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_45/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_45/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_45/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_13_45/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 2:19 - loss: 0.7046 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.2767 - mrcnn_class_loss: 0.0411 - mrcnn_bbox_loss: 0.2421 - mrcnn_mask_loss: 0.13702025-05-23 18:13:54.448253: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 2:10 - loss: 0.6962 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.2708 - mrcnn_class_loss: 0.0412 - mrcnn_bbox_loss: 0.2424 - mrcnn_mask_loss: 0.13442025-05-23 18:13:59.619675: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:13:59.624720: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10018 activity events. 2025-05-23 18:14:00.491438: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_00 2025-05-23 18:14:00.811064: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_00/marlene.trace.json.gz 2025-05-23 18:14:01.166150: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.164 ms 2025-05-23 18:14:01.351088: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_00/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 2:03 - loss: 0.6784 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.2613 - mrcnn_class_loss: 0.0399 - mrcnn_bbox_loss: 0.2374 - mrcnn_mask_loss: 0.13232025-05-23 18:14:07.786986: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 1:56 - loss: 0.6727 - rpn_class_loss: 0.0075 - rpn_bbox_loss: 0.2535 - mrcnn_class_loss: 0.0393 - mrcnn_bbox_loss: 0.2369 - mrcnn_mask_loss: 0.13552025-05-23 18:14:15.026007: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:14:15.037927: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9988 activity events. 2025-05-23 18:14:16.020567: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_15 2025-05-23 18:14:16.318214: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_15/marlene.trace.json.gz 2025-05-23 18:14:16.777296: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.271 ms 2025-05-23 18:14:17.002875: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_15Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_15/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_15/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_15/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_15/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 1:51 - loss: 0.6657 - rpn_class_loss: 0.0073 - rpn_bbox_loss: 0.2514 - mrcnn_class_loss: 0.0388 - mrcnn_bbox_loss: 0.2331 - mrcnn_mask_loss: 0.13502025-05-23 18:14:26.131317: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 1:43 - loss: 0.6565 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.2485 - mrcnn_class_loss: 0.0378 - mrcnn_bbox_loss: 0.2286 - mrcnn_mask_loss: 0.13432025-05-23 18:14:32.733541: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:14:32.740806: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9980 activity events. 2025-05-23 18:14:33.554564: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_33 2025-05-23 18:14:33.844118: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_33/marlene.trace.json.gz 2025-05-23 18:14:34.126791: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.405 ms 2025-05-23 18:14:34.339016: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_33Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_33/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_33/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_33/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_33/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 1:36 - loss: 0.6784 - rpn_class_loss: 0.0073 - rpn_bbox_loss: 0.2614 - mrcnn_class_loss: 0.0370 - mrcnn_bbox_loss: 0.2365 - mrcnn_mask_loss: 0.13622025-05-23 18:14:40.025865: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 1:28 - loss: 0.7104 - rpn_class_loss: 0.0075 - rpn_bbox_loss: 0.2767 - mrcnn_class_loss: 0.0368 - mrcnn_bbox_loss: 0.2515 - mrcnn_mask_loss: 0.13792025-05-23 18:14:47.211149: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:14:47.221347: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9954 callback api events and 9981 activity events. 2025-05-23 18:14:48.640821: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_48 2025-05-23 18:14:49.076201: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_48/marlene.trace.json.gz 2025-05-23 18:14:49.680083: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 11.506 ms 2025-05-23 18:14:50.036876: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_48Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_48/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_48/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_48/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_14_48/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 1:20 - loss: 0.7308 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.2751 - mrcnn_class_loss: 0.0367 - mrcnn_bbox_loss: 0.2700 - mrcnn_mask_loss: 0.14142025-05-23 18:14:53.179159: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 1:13 - loss: 0.7447 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.2700 - mrcnn_class_loss: 0.0369 - mrcnn_bbox_loss: 0.2851 - mrcnn_mask_loss: 0.14502025-05-23 18:15:01.670491: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:15:01.675924: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10030 activity events. 2025-05-23 18:15:02.529490: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_02 2025-05-23 18:15:02.885108: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_02/marlene.trace.json.gz 2025-05-23 18:15:03.225231: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.497 ms 2025-05-23 18:15:03.394351: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_02Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_02/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_02/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_02/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_02/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 1:05 - loss: 0.7466 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.2741 - mrcnn_class_loss: 0.0361 - mrcnn_bbox_loss: 0.2857 - mrcnn_mask_loss: 0.14292025-05-23 18:15:06.565889: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 59s - loss: 0.7470 - rpn_class_loss: 0.0079 - rpn_bbox_loss: 0.2767 - mrcnn_class_loss: 0.0353 - mrcnn_bbox_loss: 0.2862 - mrcnn_mask_loss: 0.1409 2025-05-23 18:15:17.700356: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:15:17.715353: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9972 activity events. 2025-05-23 18:15:18.675810: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_18 2025-05-23 18:15:19.004585: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_18/marlene.trace.json.gz 2025-05-23 18:15:19.282996: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.002 ms 2025-05-23 18:15:19.445219: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_18Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_18/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_18/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_18/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_18/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 50s - loss: 0.7391 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.2737 - mrcnn_class_loss: 0.0348 - mrcnn_bbox_loss: 0.2842 - mrcnn_mask_loss: 0.13862025-05-23 18:15:19.925045: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 43s - loss: 0.7299 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.2694 - mrcnn_class_loss: 0.0349 - mrcnn_bbox_loss: 0.2815 - mrcnn_mask_loss: 0.13652025-05-23 18:15:29.762734: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:15:29.768805: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10039 activity events. 2025-05-23 18:15:30.688357: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_30 2025-05-23 18:15:31.015800: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_30/marlene.trace.json.gz 2025-05-23 18:15:31.270346: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.688 ms 2025-05-23 18:15:31.448085: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_30Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_30/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_30/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_30/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_30/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 36s - loss: 0.7212 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.2663 - mrcnn_class_loss: 0.0349 - mrcnn_bbox_loss: 0.2779 - mrcnn_mask_loss: 0.13482025-05-23 18:15:37.499389: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 29s - loss: 0.7119 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.2632 - mrcnn_class_loss: 0.0345 - mrcnn_bbox_loss: 0.2739 - mrcnn_mask_loss: 0.13302025-05-23 18:15:47.585564: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:15:47.598020: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10001 activity events. 2025-05-23 18:15:48.864058: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_48 2025-05-23 18:15:49.260813: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_48/marlene.trace.json.gz 2025-05-23 18:15:49.650059: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.519 ms 2025-05-23 18:15:49.954940: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_48Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_48/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_48/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_48/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_15_48/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 22s - loss: 0.7003 - rpn_class_loss: 0.0070 - rpn_bbox_loss: 0.2577 - mrcnn_class_loss: 0.0344 - mrcnn_bbox_loss: 0.2698 - mrcnn_mask_loss: 0.13132025-05-23 18:15:52.074757: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 14s - loss: 0.6882 - rpn_class_loss: 0.0069 - rpn_bbox_loss: 0.2520 - mrcnn_class_loss: 0.0342 - mrcnn_bbox_loss: 0.2653 - mrcnn_mask_loss: 0.12972025-05-23 18:16:01.351986: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:16:01.357915: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10001 activity events. 2025-05-23 18:16:02.269808: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_01 2025-05-23 18:16:02.659793: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_01/marlene.trace.json.gz 2025-05-23 18:16:03.119441: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.347 ms 2025-05-23 18:16:03.484711: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_01Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_01/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_01/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_01/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_01/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 7s - loss: 0.6795 - rpn_class_loss: 0.0068 - rpn_bbox_loss: 0.2474 - mrcnn_class_loss: 0.0346 - mrcnn_bbox_loss: 0.2624 - mrcnn_mask_loss: 0.1283 2025-05-23 18:16:07.989906: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 18:16:16.908074: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:16:16.916136: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9958 activity events. 2025-05-23 18:16:17.683163: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_17 2025-05-23 18:16:17.974964: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_17/marlene.trace.json.gz 2025-05-23 18:16:18.235249: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.428 ms 2025-05-23 18:16:18.445051: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_17Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_17/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_17/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_17/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_17/marlene.kernel_stats.pb 40/40 [==============================] - 303s 8s/step - loss: 0.6695 - rpn_class_loss: 0.0068 - rpn_bbox_loss: 0.2429 - mrcnn_class_loss: 0.0347 - mrcnn_bbox_loss: 0.2583 - mrcnn_mask_loss: 0.1269 - val_loss: 0.5315 - val_rpn_class_loss: 0.0043 - val_rpn_bbox_loss: 0.2804 - val_mrcnn_class_loss: 0.0209 - val_mrcnn_bbox_loss: 0.1319 - val_mrcnn_mask_loss: 0.0941 Epoch 6/6 1/40 [..............................] - ETA: 18s - loss: 0.3803 - rpn_class_loss: 0.0046 - rpn_bbox_loss: 0.0827 - mrcnn_class_loss: 0.0124 - mrcnn_bbox_loss: 0.1883 - mrcnn_mask_loss: 0.09222025-05-23 18:16:25.564398: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2/40 [>.............................] - ETA: 2:04 - loss: 0.3588 - rpn_class_loss: 0.0046 - rpn_bbox_loss: 0.0817 - mrcnn_class_loss: 0.0104 - mrcnn_bbox_loss: 0.1699 - mrcnn_mask_loss: 0.09222025-05-23 18:16:31.853798: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:16:31.860296: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 9996 activity events. 2025-05-23 18:16:33.050314: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_32 2025-05-23 18:16:33.345276: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_32/marlene.trace.json.gz 2025-05-23 18:16:33.716224: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 8.479 ms 2025-05-23 18:16:33.984159: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_32Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_32/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_32/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_32/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_32/marlene.kernel_stats.pb 3/40 [=>............................] - ETA: 2:26 - loss: 0.4281 - rpn_class_loss: 0.0046 - rpn_bbox_loss: 0.1838 - mrcnn_class_loss: 0.0115 - mrcnn_bbox_loss: 0.1432 - mrcnn_mask_loss: 0.08502025-05-23 18:16:36.929961: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 4/40 [==>...........................] - ETA: 3:07 - loss: 0.4511 - rpn_class_loss: 0.0046 - rpn_bbox_loss: 0.2201 - mrcnn_class_loss: 0.0141 - mrcnn_bbox_loss: 0.1320 - mrcnn_mask_loss: 0.08022025-05-23 18:16:46.109277: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:16:46.112601: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10006 activity events. 2025-05-23 18:16:46.963985: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_46 2025-05-23 18:16:47.252277: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_46/marlene.trace.json.gz 2025-05-23 18:16:47.615767: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.947 ms 2025-05-23 18:16:47.871214: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_46Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_46/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_46/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_46/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_46/marlene.kernel_stats.pb 5/40 [==>...........................] - ETA: 2:47 - loss: 0.4234 - rpn_class_loss: 0.0043 - rpn_bbox_loss: 0.1943 - mrcnn_class_loss: 0.0147 - mrcnn_bbox_loss: 0.1319 - mrcnn_mask_loss: 0.07812025-05-23 18:16:48.993878: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 6/40 [===>..........................] - ETA: 3:09 - loss: 0.3978 - rpn_class_loss: 0.0041 - rpn_bbox_loss: 0.1755 - mrcnn_class_loss: 0.0146 - mrcnn_bbox_loss: 0.1273 - mrcnn_mask_loss: 0.07632025-05-23 18:16:58.891914: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:16:58.900393: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10034 activity events. 2025-05-23 18:16:59.709015: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_59 2025-05-23 18:16:59.895192: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_59/marlene.trace.json.gz 2025-05-23 18:17:00.248591: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.791 ms 2025-05-23 18:17:00.475387: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_59Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_59/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_59/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_59/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_16_59/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 7/40 [====>.........................] - ETA: 3:03 - loss: 0.4314 - rpn_class_loss: 0.0062 - rpn_bbox_loss: 0.1985 - mrcnn_class_loss: 0.0158 - mrcnn_bbox_loss: 0.1366 - mrcnn_mask_loss: 0.07442025-05-23 18:17:03.911737: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) /usr/local/lib/python3.8/dist-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) 8/40 [=====>........................] - ETA: 3:07 - loss: 0.5202 - rpn_class_loss: 0.0068 - rpn_bbox_loss: 0.2826 - mrcnn_class_loss: 0.0176 - mrcnn_bbox_loss: 0.1396 - mrcnn_mask_loss: 0.07362025-05-23 18:17:12.187429: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:17:12.195039: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9993 callback api events and 10033 activity events. 2025-05-23 18:17:13.173515: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_12 2025-05-23 18:17:13.464816: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_12/marlene.trace.json.gz 2025-05-23 18:17:14.081126: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 9.915 ms 2025-05-23 18:17:14.385281: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_12Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_12/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_12/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_12/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_12/marlene.kernel_stats.pb 9/40 [=====>........................] - ETA: 2:58 - loss: 0.4921 - rpn_class_loss: 0.0077 - rpn_bbox_loss: 0.2588 - mrcnn_class_loss: 0.0170 - mrcnn_bbox_loss: 0.1363 - mrcnn_mask_loss: 0.07242025-05-23 18:17:16.948024: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 10/40 [======>.......................] - ETA: 3:00 - loss: 0.4871 - rpn_class_loss: 0.0073 - rpn_bbox_loss: 0.2579 - mrcnn_class_loss: 0.0178 - mrcnn_bbox_loss: 0.1335 - mrcnn_mask_loss: 0.07062025-05-23 18:17:25.719219: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:17:25.725824: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10034 activity events. 2025-05-23 18:17:26.685872: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_26 2025-05-23 18:17:26.933774: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_26/marlene.trace.json.gz 2025-05-23 18:17:27.317593: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.636 ms 2025-05-23 18:17:27.496692: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_26Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_26/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_26/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_26/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_26/marlene.kernel_stats.pb 11/40 [=======>......................] - ETA: 2:51 - loss: 0.4781 - rpn_class_loss: 0.0068 - rpn_bbox_loss: 0.2514 - mrcnn_class_loss: 0.0174 - mrcnn_bbox_loss: 0.1314 - mrcnn_mask_loss: 0.07122025-05-23 18:17:30.200038: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 12/40 [========>.....................] - ETA: 2:54 - loss: 0.4601 - rpn_class_loss: 0.0063 - rpn_bbox_loss: 0.2340 - mrcnn_class_loss: 0.0181 - mrcnn_bbox_loss: 0.1297 - mrcnn_mask_loss: 0.07202025-05-23 18:17:41.023306: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:17:41.033175: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10055 activity events. 2025-05-23 18:17:42.030667: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_41 2025-05-23 18:17:42.557144: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_41/marlene.trace.json.gz 2025-05-23 18:17:42.953640: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.872 ms 2025-05-23 18:17:43.222764: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_41Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_41/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_41/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_41/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_41/marlene.kernel_stats.pb 13/40 [========>.....................] - ETA: 2:46 - loss: 0.4885 - rpn_class_loss: 0.0059 - rpn_bbox_loss: 0.2227 - mrcnn_class_loss: 0.0188 - mrcnn_bbox_loss: 0.1664 - mrcnn_mask_loss: 0.07482025-05-23 18:17:45.368671: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 14/40 [=========>....................] - ETA: 2:49 - loss: 0.5069 - rpn_class_loss: 0.0055 - rpn_bbox_loss: 0.2143 - mrcnn_class_loss: 0.0186 - mrcnn_bbox_loss: 0.1918 - mrcnn_mask_loss: 0.07682025-05-23 18:17:56.831457: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:17:56.839150: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 10021 activity events. 2025-05-23 18:17:57.559505: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_57 2025-05-23 18:17:57.851142: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_57/marlene.trace.json.gz 2025-05-23 18:17:58.092234: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 3.159 ms 2025-05-23 18:17:58.257739: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_57Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_57/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_57/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_57/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_17_57/marlene.kernel_stats.pb /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (2.068265). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.709774). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.686710). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.374988). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.362329). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.403444). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.963875). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.163864). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.139827). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.171992). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.293864). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.588402). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.045007). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.048363). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.215047). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (2.229152). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.446387). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.291911). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.294678). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.637283). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.639476). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.760237). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.240375). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.282264). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (2.745070). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.820864). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.961857). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.975934). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.973829). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.962804). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.051522). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.056901). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.054389). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.984673). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.983945). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.052526). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.027212). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.214696). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.070812). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.140992). Check your callbacks. warnings.warn( 15/40 [==========>...................] - ETA: 2:36 - loss: 0.5360 - rpn_class_loss: 0.0058 - rpn_bbox_loss: 0.2052 - mrcnn_class_loss: 0.0314 - mrcnn_bbox_loss: 0.2041 - mrcnn_mask_loss: 0.08942025-05-23 18:17:58.772571: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 16/40 [===========>..................] - ETA: 2:36 - loss: 0.5648 - rpn_class_loss: 0.0054 - rpn_bbox_loss: 0.2009 - mrcnn_class_loss: 0.0440 - mrcnn_bbox_loss: 0.2139 - mrcnn_mask_loss: 0.10062025-05-23 18:18:09.493322: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:18:09.499917: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10049 activity events. 2025-05-23 18:18:10.365488: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_09 2025-05-23 18:18:10.643380: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_09/marlene.trace.json.gz 2025-05-23 18:18:10.903528: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.651 ms 2025-05-23 18:18:11.064672: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_09Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_09/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_09/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_09/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_09/marlene.kernel_stats.pb 17/40 [===========>..................] - ETA: 2:24 - loss: 0.5555 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.1901 - mrcnn_class_loss: 0.0434 - mrcnn_bbox_loss: 0.2146 - mrcnn_mask_loss: 0.10162025-05-23 18:18:11.691486: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 18/40 [============>.................] - ETA: 2:23 - loss: 0.5443 - rpn_class_loss: 0.0060 - rpn_bbox_loss: 0.1804 - mrcnn_class_loss: 0.0430 - mrcnn_bbox_loss: 0.2127 - mrcnn_mask_loss: 0.10232025-05-23 18:18:23.004309: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:18:23.008750: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10022 activity events. 2025-05-23 18:18:23.844407: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_23 2025-05-23 18:18:24.105738: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_23/marlene.trace.json.gz 2025-05-23 18:18:24.518891: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.058 ms 2025-05-23 18:18:24.729475: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_23Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_23/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_23/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_23/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_23/marlene.kernel_stats.pb 19/40 [=============>................] - ETA: 2:12 - loss: 0.5317 - rpn_class_loss: 0.0058 - rpn_bbox_loss: 0.1764 - mrcnn_class_loss: 0.0415 - mrcnn_bbox_loss: 0.2069 - mrcnn_mask_loss: 0.10102025-05-23 18:18:25.311548: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 20/40 [==============>...............] - ETA: 2:08 - loss: 0.5184 - rpn_class_loss: 0.0058 - rpn_bbox_loss: 0.1684 - mrcnn_class_loss: 0.0402 - mrcnn_bbox_loss: 0.2035 - mrcnn_mask_loss: 0.10052025-05-23 18:18:34.142494: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:18:34.150363: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 10010 activity events. 2025-05-23 18:18:35.116767: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_34 2025-05-23 18:18:35.395437: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_34/marlene.trace.json.gz 2025-05-23 18:18:35.703756: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.988 ms 2025-05-23 18:18:35.990575: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_34/marlene.kernel_stats.pb 21/40 [==============>...............] - ETA: 2:01 - loss: 0.5085 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.1646 - mrcnn_class_loss: 0.0392 - mrcnn_bbox_loss: 0.1984 - mrcnn_mask_loss: 0.10072025-05-23 18:18:39.253512: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 22/40 [===============>..............] - ETA: 1:55 - loss: 0.5133 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.1764 - mrcnn_class_loss: 0.0379 - mrcnn_bbox_loss: 0.1941 - mrcnn_mask_loss: 0.09922025-05-23 18:18:46.310056: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:18:46.314858: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9918 callback api events and 9962 activity events. 2025-05-23 18:18:47.390667: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_46 2025-05-23 18:18:47.690302: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_46/marlene.trace.json.gz 2025-05-23 18:18:48.121054: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 7.689 ms 2025-05-23 18:18:48.373937: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_46Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_46/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_46/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_46/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_18_46/marlene.kernel_stats.pb 23/40 [================>.............] - ETA: 1:48 - loss: 0.5012 - rpn_class_loss: 0.0057 - rpn_bbox_loss: 0.1695 - mrcnn_class_loss: 0.0377 - mrcnn_bbox_loss: 0.1895 - mrcnn_mask_loss: 0.09872025-05-23 18:18:52.282780: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 24/40 [=================>............] - ETA: 1:42 - loss: 0.5495 - rpn_class_loss: 0.0058 - rpn_bbox_loss: 0.1974 - mrcnn_class_loss: 0.0385 - mrcnn_bbox_loss: 0.2023 - mrcnn_mask_loss: 0.10552025-05-23 18:18:59.816512: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:18:59.819281: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10038 activity events. 2025-05-23 18:19:00.650421: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_00 2025-05-23 18:19:00.947995: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_00/marlene.trace.json.gz 2025-05-23 18:19:01.283891: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.29 ms 2025-05-23 18:19:01.478327: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_00Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_00/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_00/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_00/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_00/marlene.kernel_stats.pb 25/40 [=================>............] - ETA: 1:36 - loss: 0.5509 - rpn_class_loss: 0.0059 - rpn_bbox_loss: 0.2070 - mrcnn_class_loss: 0.0376 - mrcnn_bbox_loss: 0.1967 - mrcnn_mask_loss: 0.10382025-05-23 18:19:06.012887: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 26/40 [==================>...........] - ETA: 1:28 - loss: 0.5819 - rpn_class_loss: 0.0060 - rpn_bbox_loss: 0.2121 - mrcnn_class_loss: 0.0386 - mrcnn_bbox_loss: 0.2169 - mrcnn_mask_loss: 0.10832025-05-23 18:19:10.841469: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:19:10.853866: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10016 activity events. 2025-05-23 18:19:11.854133: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_11 2025-05-23 18:19:12.173829: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_11/marlene.trace.json.gz 2025-05-23 18:19:12.623184: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.498 ms 2025-05-23 18:19:12.915828: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_11Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_11/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_11/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_11/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_11/marlene.kernel_stats.pb 27/40 [===================>..........] - ETA: 1:23 - loss: 0.5961 - rpn_class_loss: 0.0062 - rpn_bbox_loss: 0.2178 - mrcnn_class_loss: 0.0389 - mrcnn_bbox_loss: 0.2213 - mrcnn_mask_loss: 0.11182025-05-23 18:19:19.524702: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 28/40 [====================>.........] - ETA: 1:15 - loss: 0.5990 - rpn_class_loss: 0.0067 - rpn_bbox_loss: 0.2130 - mrcnn_class_loss: 0.0380 - mrcnn_bbox_loss: 0.2295 - mrcnn_mask_loss: 0.11182025-05-23 18:19:20.481323: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:19:20.488100: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 9967 activity events. 2025-05-23 18:19:21.465822: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_21 2025-05-23 18:19:21.748808: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_21/marlene.trace.json.gz 2025-05-23 18:19:22.083999: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.459 ms 2025-05-23 18:19:22.265806: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_21Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_21/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_21/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_21/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_21/marlene.kernel_stats.pb 29/40 [====================>.........] - ETA: 1:11 - loss: 0.6107 - rpn_class_loss: 0.0068 - rpn_bbox_loss: 0.2106 - mrcnn_class_loss: 0.0388 - mrcnn_bbox_loss: 0.2393 - mrcnn_mask_loss: 0.11522025-05-23 18:19:33.868942: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 30/40 [=====================>........] - ETA: 1:03 - loss: 0.6098 - rpn_class_loss: 0.0066 - rpn_bbox_loss: 0.2128 - mrcnn_class_loss: 0.0381 - mrcnn_bbox_loss: 0.2376 - mrcnn_mask_loss: 0.11472025-05-23 18:19:34.746468: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:19:34.751441: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 9967 activity events. 2025-05-23 18:19:35.526747: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_35 2025-05-23 18:19:35.847759: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_35/marlene.trace.json.gz 2025-05-23 18:19:36.328938: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 10.02 ms 2025-05-23 18:19:36.595027: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_35Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_35/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_35/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_35/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_35/marlene.kernel_stats.pb 31/40 [======================>.......] - ETA: 57s - loss: 0.6035 - rpn_class_loss: 0.0071 - rpn_bbox_loss: 0.2078 - mrcnn_class_loss: 0.0370 - mrcnn_bbox_loss: 0.2371 - mrcnn_mask_loss: 0.1145 2025-05-23 18:19:41.711079: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 32/40 [=======================>......] - ETA: 50s - loss: 0.5948 - rpn_class_loss: 0.0075 - rpn_bbox_loss: 0.2039 - mrcnn_class_loss: 0.0362 - mrcnn_bbox_loss: 0.2336 - mrcnn_mask_loss: 0.11352025-05-23 18:19:49.350712: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:19:49.364815: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10042 activity events. 2025-05-23 18:19:50.656995: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_50 2025-05-23 18:19:51.091442: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_50/marlene.trace.json.gz 2025-05-23 18:19:51.617405: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 12.207 ms 2025-05-23 18:19:52.002737: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_50Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_50/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_50/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_50/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_19_50/marlene.kernel_stats.pb 33/40 [=======================>......] - ETA: 44s - loss: 0.5898 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.2038 - mrcnn_class_loss: 0.0358 - mrcnn_bbox_loss: 0.2298 - mrcnn_mask_loss: 0.11312025-05-23 18:19:56.830924: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 34/40 [========================>.....] - ETA: 38s - loss: 0.5908 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.1991 - mrcnn_class_loss: 0.0375 - mrcnn_bbox_loss: 0.2314 - mrcnn_mask_loss: 0.11542025-05-23 18:20:03.492073: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:20:03.499654: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9994 callback api events and 10030 activity events. 2025-05-23 18:20:04.361897: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_03 2025-05-23 18:20:04.696107: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_03/marlene.trace.json.gz 2025-05-23 18:20:05.040479: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 6.123 ms 2025-05-23 18:20:05.261258: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_03Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_03/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_03/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_03/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_03/marlene.kernel_stats.pb 35/40 [=========================>....] - ETA: 32s - loss: 0.5838 - rpn_class_loss: 0.0078 - rpn_bbox_loss: 0.1971 - mrcnn_class_loss: 0.0366 - mrcnn_bbox_loss: 0.2277 - mrcnn_mask_loss: 0.11452025-05-23 18:20:10.398150: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 36/40 [==========================>...] - ETA: 25s - loss: 0.5909 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.2100 - mrcnn_class_loss: 0.0360 - mrcnn_bbox_loss: 0.2244 - mrcnn_mask_loss: 0.11292025-05-23 18:20:19.427589: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:20:19.444258: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 10003 activity events. 2025-05-23 18:20:20.266166: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_19 2025-05-23 18:20:20.520553: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_19/marlene.trace.json.gz 2025-05-23 18:20:20.838269: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.29 ms 2025-05-23 18:20:21.047589: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_19Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_19/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_19/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_19/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_19/marlene.kernel_stats.pb 37/40 [==========================>...] - ETA: 19s - loss: 0.5875 - rpn_class_loss: 0.0076 - rpn_bbox_loss: 0.2055 - mrcnn_class_loss: 0.0373 - mrcnn_bbox_loss: 0.2224 - mrcnn_mask_loss: 0.11482025-05-23 18:20:21.608964: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 38/40 [===========================>..] - ETA: 13s - loss: 0.5853 - rpn_class_loss: 0.0074 - rpn_bbox_loss: 0.2062 - mrcnn_class_loss: 0.0366 - mrcnn_bbox_loss: 0.2210 - mrcnn_mask_loss: 0.11402025-05-23 18:20:34.241007: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:20:34.245048: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9956 callback api events and 10028 activity events. 2025-05-23 18:20:35.103019: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_34 2025-05-23 18:20:35.409030: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_34/marlene.trace.json.gz 2025-05-23 18:20:35.756520: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 4.174 ms 2025-05-23 18:20:35.893227: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_34Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_34/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_34/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_34/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_34/marlene.kernel_stats.pb 39/40 [============================>.] - ETA: 6s - loss: 0.5874 - rpn_class_loss: 0.0072 - rpn_bbox_loss: 0.2134 - mrcnn_class_loss: 0.0364 - mrcnn_bbox_loss: 0.2178 - mrcnn_mask_loss: 0.1125 2025-05-23 18:20:36.504597: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started. 2025-05-23 18:20:51.487257: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1479] CUPTI activity buffer flushed 2025-05-23 18:20:51.492152: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:216] GpuTracer has collected 9955 callback api events and 10046 activity events. 2025-05-23 18:20:52.432451: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_52 2025-05-23 18:20:52.741665: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_52/marlene.trace.json.gz 2025-05-23 18:20:53.092280: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 5.379 ms 2025-05-23 18:20:53.319255: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_52Dumped tool data for overview_page.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_52/marlene.overview_page.pb Dumped tool data for input_pipeline.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_52/marlene.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_52/marlene.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to /data/data_root/learn_mask_pancarte_230525/learn_mask_pancarte_23052520250523T1749/train/plugins/profile/2025_05_23_18_20_52/marlene.kernel_stats.pb 40/40 [==============================] - 273s 7s/step - loss: 0.5896 - rpn_class_loss: 0.0071 - rpn_bbox_loss: 0.2155 - mrcnn_class_loss: 0.0364 - mrcnn_bbox_loss: 0.2172 - mrcnn_mask_loss: 0.1133 - val_loss: 0.4373 - val_rpn_class_loss: 0.0042 - val_rpn_bbox_loss: 0.2169 - val_mrcnn_class_loss: 0.0153 - val_mrcnn_bbox_loss: 0.1205 - val_mrcnn_mask_loss: 0.0804 /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.070077). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.970346). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.968883). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.968901). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.970194). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (0.998196). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.052998). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.061816). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.073372). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.150693). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.204064). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.212509). Check your callbacks. warnings.warn( /usr/local/lib/python3.8/dist-packages/keras/callbacks/callbacks.py:92: RuntimeWarning: Method (on_train_batch_end) is slow compared to the batch update (1.145670). Check your callbacks. warnings.warn( model_param file didn't exist model didn't exist , save the new model in s3 local folder to save in s3 : /data/models_weight/learn_mask_pancarte_230525 update : 1748017264.1583517 done ! 1748017268.7155173 {'files': [{'name': 'mask_model.h5', 'size': 255878432, 'last_modified': '2025-05-23T16:21:04.212260', 'hash': 'ba258950b444388c0a5ff9b0600a1555'}], 'directories': []} create thcl : begin to create parametres in sql : time : 2025-05-23 18:21:08 insert information in MTRPhoto.photo_desc_type_params: INSERT INTO MTRPhoto.photo_desc_type_params (name, svm_number_of_descriptors, size_desc, caffemodel_name, layer_name, scale_desc, nb_discretization, crop_stripes, nb_bit, shift_desc, default_storage_type, created_at, updated_at) VALUES ('learn_mask_pancarte_230525', '16384', '25088', 'learn_mask_pancarte_230525', 'pool5', '10', '256', '0', '8', '-1000', '3', '2025-05-23 18:21:08', '2025-05-23 18:21:08'); insert le photo_hashtag_type: INSERT INTO MTRBack.photo_hashtag_type (id, type) VALUES ('5009', 'learn_mask_pancarte_230525'); insert information in MTRDatou.classification_theme: INSERT INTO MTRDatou.classification_theme (mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification) VALUES ('31', 'learn_mask_pancarte_230525', '0', '0,0,0', 'background,badge,pancarte', '0,0,0', '5009', '6105', '10000', '1000', 0, 0, '2025-05-23 18:21:08', '2025-05-23 18:21:08', 'mask_rcnn'); photo_hashtag_type id: 5009 photo_desc_type id: 6105 manual tagging problem id: 3911 insert informations in datou : thcl : 3911 insert model in mtr_datou : insert step in MTRPhoto.mtr_datou_step datou_id : 4940 (step id 14091) done! UPDATE MTRDatou.mtr_datou_step SET param_json = '{"photo_hashtag_type":5009}' WHERE mtd_id = '4940'AND type = 1; # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! execution d'update_param_json param_json_actuel: ((None,),) self.json_param: {} new_param_json: {"pht": 5007} UPDATE MTRDatou.mtr_datou_step SET param_json='{"pht": 5007}' WHERE id=14091; fin d'execution d'update_param_json 1312.25user 5658.51system 33:10.76elapsed 350%CPU (0avgtext+0avgdata 4283496maxresident)k 7343576inputs+6497072outputs (249897major+106287350minor)pagefaults 0swaps