FWSM 10 Gig performance issues

[lap]

Hi all,

Our customer is experimenting really bad performance when runing 10Gig traffic through FWSM on C6509. Test with1 Gig traffic are providing find result perfromance as expected in this document: https://supportforums.cisco.com/docs/DOC-12668. I have made a simple drawing so everyone can understand the setup:

The issue is when running 10 Gig traffic between Netapp servers. This traffic is going though the FWSM and the perfomance are really bad: around 50 Mbit/sec. If the traffic is not going though the FWSM ther performance are around 900 Mbit/s.

The customer and I think that the issue is releated the buffer in the C6509 and the FWSM which has big trouble managing 10G to 1G traffic convertering between C6509 and FWSM 6 G etherchannel connection.

When running 10G traffic through FWSM the number of output drops are increasing as you can see on the output bellow. The last thing which is wired a is that the speed is showing 1000 Mbits and not 6000Mbits :

****************************************************************************************************************************************

RSD-ODE-ED2#show firewall module 9 traffic

Firewall module 9:

Specified interface is up line protocol is up (connected)

  Hardware is EtherChannel, address is 001e.1356.5fcb (bia 001e.1356.5fcf)

  MTU 1500 bytes, BW 6000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 31/255, rxload 31/255

  Encapsulation ARPA, loopback not set

  Full-duplex, 1000Mb/s, media type is unknown

  input flow-control is on, output flow-control is on

  Members in this channel: Gi9/1 Gi9/2 Gi9/3 Gi9/4 Gi9/5 Gi9/6

  Last input never, output never, output hang never

  Last clearing of "show interface" counters 2d23h

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 43422

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 732991000 bits/sec, 130541 packets/sec

  5 minute output rate 732572000 bits/sec, 128282 packets/sec

     21893539137 packets input, 18804993733769 bytes, 0 no buffer

     Received 127691643 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 input packets with dribble condition detected

     21722886031 packets output, 18888822170227 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

****************************************************************************************************************************************

Best regards,

Laurent

[mirober2]

Hi Laurent,

Unfortunately, the FWSM is simply not capable of processing 10 Gbps of traffic. As you noted, the port-channel between the FWSM and the 6500 backplane is only a 6 Gbps bundle, so even under the most ideal conditions it will not be capable of handling 10 Gbps.

If this level of throughput is required for your environment, I would recommend talking with your Cisco account team or partner about the ASA or ASA-SM platforms and get some design assistance from them to integrate those into your network.

Hope that helps.

-Mike

[lap]

Hi Mike,

The thing is when running "normal" TCP traffic through FWSM performance are good (around 600 Mbit/s) but as soon as we run Netapp traffic through FWSM the performance are really realyy low (around 13 Mbit/s). So there is something wrong with the Netapp traffic.

Best regards,

Laurent

[Marcin Latosiewicz]

Laurent,

FWSM has several tweak you can do to improve single flow TCP performance.

There's great article by Andrew Ossipov:

https://supportforums.cisco.com/docs/DOC-12668

Among the things you should try:

- enabling NP completion unit

- disabling SACK through FWSM

A sniffer trace of traffic would be great, but analyzing it is a bit of a chore and if it comes to this I would suggest opening a TAC case.

HTH,

Marcin

[councilm]

I will second Marcin's recommendation.  That document is very useful and I was seeing similar performance on our EMC NAS replications. 

Note: depending on FWSM OS version, the NP completion unit setting can be turned off but the config shows it as enabled.  (bug ID: CSCth72685) 

Cory C.