Re: 3750X Poor Performance - Page 2

Ryan Fisher · ‎10-19-2018

I have a bit of an odd situation. At my DR site, I used to have 2 3560 switches that were port-channeled together. I recently swapped those switches out for 2 3750X switches in a stack, and copied the identical configuration on them as the original 3560 switches.

I have a 1gb WAN connection from my main site that I use mostly for my SAN replication, and with the old 3560 switches, I was able to max out that circuit and push almost a whole 1gb bandwidth. After this swap with the new 3750X switches, I can't get it to pass more than 200mb on that port. Like I said, these switches have the same config on them, so there's nothing new there. There is also no QoS either. I've checked ports for errors, and there are none and they are negotiated properly at 1000-full. I'm out of ideas on things to check, and would greatly appreciate any guidance of things I could look at.

Thanks!

drcore01-3750x#sh int gi1/0/48
GigabitEthernet1/0/48 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 6c20.564d.4ab0 (bia 6c20.564d.4ab0)
  Description: cox 1gb metroE
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 2/255, rxload 49/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 142
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 195645000 bits/sec, 17395 packets/sec
  5 minute output rate 11115000 bits/sec, 12417 packets/sec
     317861817 packets input, 444836613365 bytes, 0 no buffer
     Received 164662 broadcasts (161104 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 161104 multicast, 0 pause input
     0 input packets with dribble condition detected
     237971247 packets output, 40530223909 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

drcore01-3750x#sh run int gi1/0/48
Building configuration...

Current configuration : 190 bytes
!
interface GigabitEthernet1/0/48
 description cox 1gb metroE
 switchport trunk allowed vlan 1,15,501,521,920,980
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

Switch Ports Model                     SW Version            SW Image                 
------ ----- -----                     ----------            ----------               
*    1 54    WS-C3750X-48              15.2(4)E6             C3750E-UNIVERSALK9-M     
     2 54    WS-C3750X-48              15.2(4)E6             C3750E-UNIVERSALK9-M

Georg Pauwen · ‎10-23-2018

There is another thread on here with somebody having the exact same problem on a 3750X. Buffer and queue tuning apparently gained him 50MB in throughput , which really isn't much...

Curious to know if you get more...

Joseph W. Doherty · ‎10-23-2018

I believe the 3750X (and the 3750E) is supposed to be wire-speed capable on all ports, concurrently. However, I also believe, that's not accounting for bursting issues.

Georg, could you post the link to the other thread?

Georg Pauwen · ‎10-23-2018

My bad, it's a 2960X. Interestingly enough, the queue looks almost identical, and no packets are dropped...yet throughput is really low...

https://community.cisco.com/t5/routing/transfert-rate-issue-2960x-10gb-lt-gt-quanta-10gb-max-1gb-why/m-p/3729300

Abzal · ‎10-23-2018

Usually tuning QoS helps. Have you tested bandwidth with iperf like software. In our corp network I have 3750X with version 15.0(2)SE7 which perfectly handling 5-7Gbit traffic. You can try to upgrade it.

Best regards,
Abzal

Ryan Fisher · ‎10-23-2018

If it's an inherent problem with 3x switches, my 3560 i had in there previously didn't have any of those problems, and I didn't have QoS enabled either. I'll enable QoS if I have to, that's not a big deal, but just puzzling.

The traffic is basically SAN replication traffic and exchange DAG replication traffic. Not much else traverses that link of any significance, so I wouldn't expect to see any microburst traffic.

Thanks for that link to the other guy's problem. I'll check that out and see if any of that relates. If I have to enable QoS, I may need some help in tuning that for maximum benefit.

Thanks!

Ryan Fisher · ‎10-23-2018

If it's an inherent problem with 3x switches, my 3560 i had in there previously didn't have any of those problems, and I didn't have QoS enabled either. I'll enable QoS if I have to, that's not a big deal, but just puzzling.

The traffic is basically SAN replication traffic and exchange DAG replication traffic. Not much else traverses that link of any significance, so I wouldn't expect to see any microburst traffic.

Thanks for that link to the other guy's problem. I'll check that out and see if any of that relates. If I have to enable QoS, I may need some help in tuning that for maximum benefit.

Thanks!

Georg Pauwen · ‎10-23-2018

Hello,

post the output of 'show mls qos queue-set 1'...

Ryan Fisher · ‎10-23-2018

Here you go. Thanks

drcore01-3750x#sh mls qos queue-set 1
Queueset: 1
Queue : 1 2 3 4
----------------------------------------------
buffers : 0 10 0 90
threshold1: 100 200 100 100
threshold2: 100 200 100 100
reserved : 50 50 50 50
maximum : 400 400 400 400

Georg Pauwen · ‎10-23-2018

The queue buffers have been changed, not sure why they are not showing up. Either way, do you notice a throughput improvement ?

@Ryan Fisher wrote:
Here you go. Thanks

drcore01-3750x#sh mls qos queue-set 1
Queueset: 1
Queue : 1 2 3 4
----------------------------------------------
buffers : 0 10 0 90
threshold1: 100 200 100 100
threshold2: 100 200 100 100
reserved : 50 50 50 50
maximum : 400 400 400 400

Ryan Fisher · ‎10-23-2018

I have seen no change unfortunately. As you can see in my bit rate. Thanks!

drcore01-3750x#sh int gi1/0/48
GigabitEthernet1/0/48 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 6c20.564d.4ab0 (bia 6c20.564d.4ab0)
  Description: cox 1gb metroE net to sd
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 2/255, rxload 48/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 6291
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 188967000 bits/sec, 16848 packets/sec
  5 minute output rate 9824000 bits/sec, 12294 packets/sec
     1290561610 packets input, 1805628641246 bytes, 0 no buffer
     Received 734315 broadcasts (718145 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 718145 multicast, 0 pause input
     0 input packets with dribble condition detected
     943983588 packets output, 103113927024 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Georg Pauwen · ‎10-24-2018

Hello,

just a thought: if the bit rate, or the traffic arriving at the local interface, is so low, the connection between the Cisco and the Qanta doesn't even come into play. What is the local PC/server you have connected to the Cisco, and what sort of traffic are you trying to send/download ? I have a feeling the problem is more local...

Joseph W. Doherty · ‎10-24-2018

Indeed it's puzzling, i.e. you didn't see issue with 3560, but you do with the 3750X.

Of course, the 3750X is a later generation than the original 3560/3750 series, and the 3750 series have stacking capability, which the 3560 series do not. That might impact how RAM for buffers is allocated. (Only Cisco can tell us what really differs, and they generally keep such information confidential. [For example, it was years before they published a paper that mentioned the 3560/3750 series have 2 MB RAM per 24 copper ports and also for the uplink ports.})

You mention, initially, you copied the 3560 config onto the 3750X, and so the configs were identical. However, if they are running different IOS versions, there might be some small changes made during that config copy. Further, you mention the 3560s had a port-channel but the 3750X are stacked, and if that's correct, then the configurations cannot be 100% identical. What impact that might have, if any, again, difficult to say, especially without access to proprietary information.

As to not having any microburst traffic, that's possible assuming there's no over subscription of bandwidth, end-to-end. Is that truly the case?

As to tuning for QoS, what has often worked well for me, is pushing the maximums up from their default of 400 to 1600 or 3200, lowering reserved buffers from they default of 50 to either 10 or 0 and I recall (?) pushing all the threshold settings up (but I forget to what value). Also, often with the prior changes, you don't need to adjust the buffer ratio from their default of 25.

What the foregoing does is insure both the logical and physical buffer limits are increased while reducing RAM reserved to port queues and placing it into a common pool that is shared dynamically, as needed.

Ryan Fisher · ‎10-24-2018

Hi, so, in regards to the configs being 100% identical, yes, you are correct, as they are not 100%. But, the differences only differ between what might work for the stacked 3750 vs the 3560 that was port-channeled to another 3560. What I did do, was copy all information that was pertinent to the network there to the new switch, (ie. vlans, ip addresses, etc) and applied each port config to the same corresponding ports on the new switch individually. I'll post the config off the old 3560 here and you'll see how they are similar.

Because this is san replication traffic, we were able to control the amount of bandwidth the san pushes for its replications. Judging from the past mrtg graphs with the old switch, I can see that it never maxed out the 1gb circuit, but gets close. I'll include that graph in this post as well.

The real reason I swapped these switches, is because I just upgraded my wan circuit to 2gb, so I needed a 10gb interface which the 3560 doesn't have. So I got these 3750X switches with the 10gb network module to accommodate that new speed. Because of that, this 1gb circuit is temporary until I get my provider to cut over to the new 2gb circuit. My fear is that if these switches can't push 1gb through one port on that switch, how's it even going to come close to 2gb? This may sound dumb, but could it be that the 10gb network module that's installed could be using RAM or some resources that could be affecting the rest of the switch? Seems unlikely, but who knows. There's nothing plugged in to that module yet, but it's installed in one of the switches in the stack.

I'm worrying about the 1gb circuit right now because A.) It's annoying that it doesn't work like you would expect when the older switch worked fine and B.) Because the longer it runs like this the more my replications fall behind and will be harder to catch up. I'm at the point now of thinking I might put the old switch back in just for that wan connection to get the speeds back up until I'm able to do that 2gb cutover. I'd rather get this switch working the way it should, but I feel I'm running out of options here.

Also, thank you guys for all your help and time with helping me try to figure this out. I'm not a heavy network engineer, but know enough to get the basics done.

Here's the chart from the past, mrtg is slowly wiping out my historical for whatever reason, but you can see that on the weekly and monthly charts the max is 92% of the bandwidth, and that's what it would be constant when replications were going.

Here's a good view from the other end site that's sending. Clearly a drop in throughput.

Old 3560 config

dr01-3560#sh run
Building configuration...

Current configuration : 11410 bytes
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname dr01-3560
!
boot-start-marker
boot-end-marker
!
!
enable
!

aaa new-model
!
!         
!
!
!
!
!
aaa session-id common
clock timezone PST -8 0
clock summer-time PDT recurring
system mtu routing 1500
authentication mac-move permit
ip routing
!
!
!
!
no mls qos rewrite ip dscp
!

!
!         

        quit
!
!         
!
errdisable recovery cause loopback
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
!
!
!
interface Port-channel10
 description PO to dr02-3560 10.100.199.6 PO10
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Port-channel11
 description drucs01_fi-a
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Port-channel12
 description drucs01_fi-b
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet0/1
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/2
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/3
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/4
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/5
 switchport access vlan 195
 spanning-tree portfast
!         
interface GigabitEthernet0/6
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/7
 switchport access vlan 200
 spanning-tree portfast
!
interface GigabitEthernet0/8
 switchport access vlan 200
 spanning-tree portfast
!
interface GigabitEthernet0/9
 switchport access vlan 200
 spanning-tree portfast
!
interface GigabitEthernet0/10
 switchport access vlan 200
 spanning-tree portfast
!
interface GigabitEthernet0/11
 switchport access vlan 200
 spanning-tree portfast
!         
interface GigabitEthernet0/12
 switchport access vlan 200
 spanning-tree portfast
!
interface GigabitEthernet0/13
 switchport access vlan 205
 spanning-tree portfast
!
interface GigabitEthernet0/14
 switchport access vlan 205
 spanning-tree portfast
!
interface GigabitEthernet0/15
 switchport access vlan 110
 shutdown
 spanning-tree portfast
!
interface GigabitEthernet0/16
 switchport access vlan 205
 spanning-tree portfast
!
interface GigabitEthernet0/17
 switchport access vlan 205
 spanning-tree portfast
!
interface GigabitEthernet0/18
 switchport access vlan 205
 spanning-tree portfast
!
interface GigabitEthernet0/19
 switchport access vlan 205
 spanning-tree portfast
!
interface GigabitEthernet0/20
 switchport access vlan 110
 speed 100
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet0/21
 description dresx05 vmnic0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 spanning-tree portfast trunk
!
interface GigabitEthernet0/22
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/23
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/24
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/25
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/26
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/27
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/28
 switchport access vlan 195
 spanning-tree portfast
!
interface GigabitEthernet0/29
 description po11 to drucs01fi-a
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol lacp
 channel-group 11 mode active
 spanning-tree portfast trunk
!
interface GigabitEthernet0/30
 description po11 to drucs01fi-a
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol lacp
 channel-group 11 mode active
 spanning-tree portfast trunk
!
interface GigabitEthernet0/31
 description po12 to drucs01fi-b
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol lacp
 channel-group 12 mode active
 spanning-tree portfast trunk
!
interface GigabitEthernet0/32
 description po12 to drucs01fi-b
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol lacp
 channel-group 12 mode active
 spanning-tree portfast trunk
!
interface GigabitEthernet0/33
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/34
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/35
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/36
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/37
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/38
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/39
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/40
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/41
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/42
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/43
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/44
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/45
 switchport access vlan 999
 spanning-tree portfast
!
interface GigabitEthernet0/46
 description cogent 100mb pt2pt
 no switchport
 ip address 10.253.221.2 255.255.255.252
 speed 100
 duplex full
!
interface GigabitEthernet0/47
 description to dr01-5510asa fa0/0
 no switchport
 ip address 10.100.194.1 255.255.255.252
 speed 100
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet0/48
 description cox 1gb metroE net
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,15,501,521,920,980
 switchport mode trunk
 shutdown
!
interface GigabitEthernet0/49
 no switchport
 ip address 10.100.193.1 255.255.255.248
!
interface GigabitEthernet0/50
 description PO to dr02-3560 10.100.199.6 gi0/50
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol pagp
 channel-group 10 mode desirable
!
interface GigabitEthernet0/51
!         
interface GigabitEthernet0/52
 description PO to dr02-3560 10.100.199.6 gi0/52
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-protocol pagp
 channel-group 10 mode desirable
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan15
 ip address 10.200.145.3 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 no ip mroute-cache
 shutdown
 standby 0 ip 10.200.145.1
 standby 0 priority 85
 standby 0 authentication 
!
interface Vlan100
 description dr client network
 ip address 10.100.80.1 255.255.252.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
!
interface Vlan101
 description dr admin client network
 ip address 10.100.85.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
!
interface Vlan110
 description voice server net
 ip address 10.100.10.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
!
interface Vlan195
 description iscsi net
 ip address 10.100.195.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
!
interface Vlan205
 description dr server net
 ip address 10.100.205.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
!
interface Vlan501
 description standby prod server net
 ip address 10.200.1.3 255.255.255.0
 no ip unreachables
 no ip proxy-arp
 no ip mroute-cache
 shutdown
 standby 0 ip 10.200.1.1
 standby 0 priority 85
 standby 0 authentication 
!
interface Vlan521
 description standby server management net
 ip address 10.202.1.3 255.255.255.0
 no ip unreachables
 no ip proxy-arp
 no ip mroute-cache
 shutdown
 standby 0 ip 10.202.1.1
 standby 0 priority 85
 standby 0 authentication
!
interface Vlan920
 description 
 ip address 10.92.1.3 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 no ip mroute-cache
 shutdown
 standby 0 ip 10.92.1.1
 standby 0 priority 85
 standby 0 authentication 
!
interface Vlan980
 description cox 1gb metroE 
 ip address 10.253.220.3 255.255.255.224
!
interface Vlan999
 description management net
 ip address 10.100.199.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
!
!
router eigrp 100
 network 10.0.0.0
 network 10.100.193.1 0.0.0.0
!
no ip http server
no ip http secure-server
!
!
ip route 0.0.0.0 0.0.0.0 10.100.194.2
ip route 10.254.100.0 255.255.255.0 10.100.194.2
ip route 172.16.20.0 255.255.255.0 10.201.1.1
ip route xxxxx 255.255.255.255 10.253.220.1
!
ip sla enable reaction-alerts
logging esm config
logging trap debugging
logging 10.200.1.93
logging 10.200.1.135
!
snmp-server community
snmp-server location Las Vegas DR
snmp-server contact admin 
!
!
!
!
line con 0
 exec-timeout 0 0
line vty 0 3
 session-timeout 120  output
 access-class 10 in
 exec-timeout 0 0
 privilege level 15
 length 0
 transport input all
line vty 4
 session-timeout 120  output
 access-class 10 in
 exec-timeout 5 0
 privilege level 15
 length 0 
 transport input all
line vty 5 15
 session-timeout 120  output
 access-class 10 in
 exec-timeout 5 0
 privilege level 15
 length 0
 transport input all
!
ntp authentication-key 10 md5 
ntp authenticate
ntp trusted-key 10
ntp server 10.200.199.1
end

Georg Pauwen · ‎10-24-2018

Hello,

my apologies if this a redundant question, the thread is becoming kind of long so maybe this has already been mentioned, but what device is at the other end of the trunk (the Cox side) ?

Also, just to make sure, check which switch is the root for the VLANs traversing the trunk (show spanning-tree vlan x).

Ryan Fisher · ‎10-24-2018

So, my DR site is at SwitchNap in Las Vegas. So, my connection goes:

3750x 1gb copper -> switch cross connect -> Cox Provider -> MetroE Cloud -> HQ office 3750G (currently 1gb fiber)

\> Remote Site

Here's the sh spanning-tree. I'm not sure which vlan to do, so i did 1 and 980 which are the two primarily associated with interface 1/0/48. Let me know if you need to see something else.

drcore01-3750x#sh spanning-tree vlan 1

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    32769
             Address     0022.0ca9.8900
             Cost        4
             Port        48 (GigabitEthernet1/0/48)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     6c20.564d.4a80
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/21            Desg FWD 4         128.21   P2p 
Gi1/0/48            Root FWD 4         128.48   P2p 
Po11                Desg FWD 3         128.568  P2p 
Po12                Desg FWD 3         128.576  P2p 
Po13                Desg FWD 3         128.584  P2p 
Po14                Desg FWD 3         128.592  P2p 
Gi2/0/21            Desg FWD 4         128.75   P2p 


drcore01-3750x#sh spanning-tree vlan 980

VLAN0980
  Spanning tree enabled protocol ieee
  Root ID    Priority    33748
             Address     0022.0ca9.8900
             Cost        4
             Port        48 (GigabitEthernet1/0/48)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    33748  (priority 32768 sys-id-ext 980)
             Address     6c20.564d.4a80
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/21            Desg FWD 4         128.21   P2p 
Gi1/0/48            Root FWD 4         128.48   P2p 
Po11                Desg FWD 3         128.568  P2p 
Po12                Desg FWD 3         128.576  P2p 
Po13                Desg FWD 3         128.584  P2p 
Po14                Desg FWD 3         128.592  P2p 
Gi2/0/21            Desg FWD 4         128.75   P2p

Thanks!