"pause output" on 6509 interfaces

[wilson_1234_2]

We have been seeing "pause output" on our 6509 interfaces on devices that have high traffic throughput.

I have seen two different descriptions of waht is telling the source port to pause.

My understanding is that the device that has the pause output is trying to send traffic to a device that has full buffers or is congested and the target end is telling the port to pause sending traffic.

The question is , what is telling the source port to pause, is it the 6509 port that the target device is connect to, or is it the target device itself?

[Giuseppe Larosa]

Hello Richard,

if the pause output increases it is the C6509 port that sends out pause frames to the connected device in an effort to perform flow control.

When the port senses its own dedicated input buffers are full it sends out these frames (if flow-control is enabled)

the device that receives IEEE output frames slows down its sending rate to help the other side handle the traffic.

see

http://www.ieee802.org/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdf

enabling flow control is a possible workaound for those linecards like some 48GE linecards also on C6500 that have a ratio 8:1 n:1 in general where not all ports can be line rate at the same time (the ports share the same ASIC hardware resource)

Hope to help

Giuseppe

[wilson_1234_2]

ok,

Sorry but:

So if my interface G1/1 is incrementing "pause output" counters, it is telling the device connected to G1/1 to slow down, I cannot handle this much input.

The device then slows down while the buffer clears and then picks back up?

And is this behavior because I do not have input flow control enabled?

My interfaces are configured per the default of:

input flow-control is off, output flow-control is on

Since the device connected to G1/1 is sending traffic to another device on the switch, should I turn on "input flow control"?

Would this be one "pause output" each time the device is told to slow down?

And how much would the traffic slow down?

Would I see the traffic continue to ramp up, then ramp all the way back down?

[blawrimore]

If anyone can answer these questions, I too would love to see them. I am having a lot of PAUSE output notices on an otherwise unconfigured port that is connected to my SAN and servers. I am getting high latentcy on those same servers. I think these answers would help me out a lot with my own troubleshooting.

[Giuseppe Larosa]

Hello Richard, Bryan

sorry for the late attempt to answer your questions.

>> input flow-control is off, output flow-control is on

This means the port has the capability to send out PAUSE frames

see this doc

http://www.cisco.com/en/US/docs/switches/lan/catalyst5000/catos/6.x/configuration/guide/gigabit.html#wp1019856

even if it is for catos see table 6-2 it says:

send on

Enables a local port to send pause frames to a remote port. Use send on when a remote port is set to receive on or receive desired.

We could say that default settings allow a GE port to try to defend itself from buffers overflow without listening to pause frames from the other side.

so what you see increasing output frames is not in contradiction with the port default settings

>> Since the device connected to G1/1 is sending traffic to another device on the switch, should I turn on "input flow control"?

Given the definitions in table 6-2 of above documents default settings should be enough to regulate traffic volume of conversations between the two hosts:

the other direction can be regulated by the output frames sent on the other port if necessary.

Enabling input flow control means asking the switch port to process possible input frames slowing down its TX rate.

>> Would this be one "pause output" each time the device is told to slow down?

Not sure if already implemented but the PAUSE frame format has time window inside that specifies the numbere of time intervals the other side should be silent.

Each basic time interval is the time it takes to send 512 octets on the wire.

So a single output frame could ask for multiple silence intervals.

These are very small intervals in the range of GE speed.

The 2005 specification should have a single link wide scope flow control with a pause operand. The 2007 proposal I've linked in my first post try to introduce class based Flow control.

As suggested by other collegues in other threads flow control is not the root cause of slowed poor performance but an attempt to deal with excessive traffic.

To be noted that the concept of "excessive traffic" doesn't mean the port is receiving more then 1Gbps but means that a pool of ports sharing an ASIC chip are in competition to use those resources.

For some linecard there is an 8:1 mapping of 8 ports over a 1Gbps ASIC.

For these linecards using flow-control can provide benefits.

Hope to help

Giuseppe

[cole.b]

I know this one is old, but thought I would add the below.

As Giuseppe states, the blade you have (probably WS-X6148-GE-TX) shares ASICs.  8 ports share a single 1MB buffer ASIC.  Using a blade like WS-X-6748-GE-TX as a 1MB buffer ASIC per port.

For normal user traffic, the 6148 should be fine but for any server or high throughput device (iSCSI, VMWare, etc.) would recommend the single port per ASIC.  

If you are getting  PAUSE outputs on your 6500s, you will most likely need to replace the blade with a higher performing blade.