Solved: Performance problems with old 6509

shfa09 · ‎03-05-2013

Hi all,

We have some performance problems with an old 6509. We use this switch in our datacenter, has around 50 vlans and is configured as gateway for serveral vlans and also EIGRP is configured.

The problem: Several times a day we notice increased latency, even within the same vlan. For example a ping from a server connected directly to this switch to a server in the same vlan also directly connected on this switch will show round trip times of 12ms. The latency will be there several minutes and then everything is back to normal and round trip times are below 1ms.

During the latency the cpu load is low(around 5%) and there are no new entry's in the log.

Several times there was a big file transfer going on and every connected host noticed delays but most of the times latency occured there was no increase in traffic. Does anyone know where i can find the specs for this switch about performance and max throughput? It could also be a software problem or a problem with one of the modules but don't know how to troubleshoot any further.

show version:

Cisco Internetwork Operating System Software

IOS (tm) c6sup2_rp Software (c6sup2_rp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 10:25 by ccai

Image text-base: 0x4002100C, data-base: 0x42020000

ROM: System Bootstrap, Version 12.1(11r)E1, RELEASE SOFTWARE (fc1)

BOOTLDR: c6sup2_rp Software (c6sup2_rp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

sw12 uptime is 1 year, 42 weeks, 3 days, 1 hour, 41 minutes

Time since sw12 switched to active is 1 year, 27 weeks, 1 day, 20 hours, 40 minutes

System returned to ROM by Stateful Switchover (SP by power-on)

System restarted at 08:01:20 CEST Fri May 13 2011

System image file is "disk0:c6k222-psv-mz.122-18.SXD7b.bin"

cisco WS-C6509 (R7000) processor (revision 2.0) with 227328K/34816K bytes of memory.

Processor board ID SCA0443013Y

R7000 CPU at 300Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB L3 Cache

Last reset from power-on

X.25 software, Version 3.0.0.

Bridging software.

18 Virtual Ethernet/IEEE 802.3 interface(s)

96 FastEthernet/IEEE 802.3 interface(s)

196 Gigabit Ethernet/IEEE 802.3 interface(s)

381K bytes of non-volatile configuration memory.

32768K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

asdx#sho module

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 2 Catalyst 6000 supervisor 2 (Hot) WS-X6K-SUP2-2GE SAL0802SM4Z

2 2 Catalyst 6000 supervisor 2 (Active) WS-X6K-SUP2-2GE SAL0801S1ZY

3 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX SAD0801040Z

4 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX SAD0801047B

5 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX SAD0801046M

6 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX SAD0801043W

8 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45 SAD04010JDF

9 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45 SAD03280012

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0009.1246.fd54 to 0009.1246.fd55 5.0 7.1(1) 12.2(18)SXD7 Ok

2 0009.1246.f39c to 0009.1246.f39d 5.0 7.1(1) 12.2(18)SXD7 Ok

3 000e.d79f.3d70 to 000e.d79f.3d9f 5.0 7.2(1) 8.3(0.156)RO Ok

4 000e.d79f.41f0 to 000e.d79f.421f 5.0 7.2(1) 8.3(0.156)RO Ok

5 000e.d79f.4c10 to 000e.d79f.4c3f 5.0 7.2(1) 8.3(0.156)RO Ok

6 000e.d79f.3e60 to 000e.d79f.3e8f 5.0 7.2(1) 8.3(0.156)RO Ok

8 0030.9617.1f9c to 0030.9617.1fcb 1.1 4.2(0.24)VAI 8.3(0.156)RO Ok

9 00d0.d331.0428 to 00d0.d331.0457 1.1 4.2(0.24)VAI 8.3(0.156)RO Ok

Mod Sub-Module Model Serial Hw Status

--- --------------------------- ------------------ ------------ ------- -------

1 Policy Feature Card 2 WS-F6K-PFC2 SAL0803SV97 3.4 Ok

1 Cat6k MSFC 2 daughterboard WS-F6K-MSFC2 SAL0751QRZ8 2.6 Ok

2 Policy Feature Card 2 WS-F6K-PFC2 SAL0801RXQR 3.4 Ok

2 Cat6k MSFC 2 daughterboard WS-F6K-MSFC2 SAL0802SKU0 2.6 Ok

Mod Online Diag Status

--- -------------------

1 Pass

2 Pass

3 Pass

4 Pass

5 Pass

6 Pass

8 Pass

9 Pass

Giuseppe Larosa · ‎03-05-2013

Hello Danny,

the performance of the WS-X6148 linecards are poor for modern datacenter: groups of 6-8 ports are served by a single ASIC chip that provides 1 Gbps per direction as aggregate volume to the group of ports.

The whole linecard connects to the shared bus at 8 Gbps per direction.

This is probably the root cause of your issues as modern servers are all capable of 1GE full speed.

There is no workaround for this. You have also sup2 with MSFC2.

Hope to help

Giuseppe

View solution in original post

Giuseppe Larosa · ‎03-05-2013

Hello Danny,

the performance of the WS-X6148 linecards are poor for modern datacenter: groups of 6-8 ports are served by a single ASIC chip that provides 1 Gbps per direction as aggregate volume to the group of ports.

The whole linecard connects to the shared bus at 8 Gbps per direction.

This is probably the root cause of your issues as modern servers are all capable of 1GE full speed.

There is no workaround for this. You have also sup2 with MSFC2.

Hope to help

Giuseppe

shfa09 · ‎03-05-2013

Thanks for the reply's! Very helpfull!

There was not a lot information i could find about this and couldn't find the specs. The only documentation i could find was http://www.cisco.com/en/US/products/hw/switches/ps708/products_module_configuration_guide_chapter09186a0080394e16.html#wp1049787 which states that the WS-X6148-GE-TX has a 32Gbps bus backplane connection as JosephDoherty also mentioned

Where did you guys find the information?

I guess it is time for a replacement, for now we will migrate the servers that generate most of the traffic to different switches.

Joseph W. Doherty · ‎03-06-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Where did you guys find the information?

The datasheets (e.g. see http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/product_data_sheet0900aecd8017376e.html) often provide the specifics for a particular line card.

Cisco's whitepaper, for the 6500 architecture (see http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html), helps it make sense.

shfa09 · ‎03-07-2013

Thanks!

Joseph W. Doherty · ‎03-05-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

As Giuseppe already noted, 6148 line cards are not designed for high performance. All your line cards are "classic bus", which means they also all share a 16 Gbps (duplex) bus. The sup2 offers 15 Mpps forwarding performance, which supports about 10 Gbps (aggregate) bandwidth for worst case Ethernet, also across the whole chassis.

As Giuseppe also noted, the 6148 line card has ports controlled by the same ASIC. You can (somewhat) avoid bottlenecking particular ASICs by how you connect ports on these line cards. Ideally you want to connect ports such that each port group has about the same usage.

If the bus or supverisor is being stressed by a limited set of very busy hosts (of which their bulk traffic is between themselves), the least expensive option might be to move those ports to a separate stand-alone switch that supports full gig wire-speed, for example a 2960-S, and connect that to the 6509.