Gigabit Etherchannel and Flow control

[chris.smailes]

I wonder if anyone can help ?

The setup is:

A 6509 running IOS (Version 12.2(18)SXD7) with two gigabit ports in an etherchannel connected to a 'Network Appliance' server using Intel NICs, teamed for load balancing.

The query is as follows:

A server administrator approached me and said that his NIC stats showed a small number of transmit queue overflows and he had noticed that the flow control status on the gigabit interfaces had autonegotiated to 'receive' even though he had set it to 'full'.

Since it is his TRANSMIT queue which is overflowing I do not think it is anything to do with the switch and is probably a configuration issue on the server but agreed to investigate.

The ports were set by default as follows:

Full-duplex, 1000Mb/s

input flow-control is off, output flow-control is on

HOWEVER, now I have set them as follows:

!

interface GigabitEthernet8/1

description FAWDON-1-V142-UNN-P3

no ip address

speed 1000

duplex full

flowcontrol receive desired

switchport

switchport access vlan 142

switchport mode access

channel-group 3 mode on

end

AND

!

interface GigabitEthernet9/1

description FAWDON-2-V142-UNN-P3

no ip address

speed 1000

duplex full

flowcontrol receive desired

switchport

switchport access vlan 142

switchport mode access

channel-group 3 mode on

end

The port channel is set as follows:

!

interface Port-channel3

description FAWDON-V142-UNN-P3

no ip address

switchport

switchport access vlan 142

switchport mode access

end

SHOW FLOWCONTROL reveals the following:

C72033-SF1#sh flowcontrol

Port Send FlowControl Receive FlowControl RxPause TxPause

admin oper admin oper

.

.

Gi8/1 desired on desired desired 0 0

.

.

Gi9/1 desired on desired desired 0 0

.

.

.

Po3 Unsupp. Unsupp. desired off 0 0

MY QUESTION is ? why does the operational status of the flow control on the physical GE ports show as desired and not on ?

(The server admin reports that his NICs now say ?full?

AND, is it necessary to change the configuration of the port-channel to enable the flow control for the physical ports.

(I can?t just try it in case the etherchannel goes into a suspended state and knocks the server offline.)

I haven?t been able to find anything in the documentation about this.

Thanks for your time,

Chris.

[jackyoung]

First of all, you have to ensure the server NIC can support etherchannel, otherwise, don't do it.

Moreover, I never heard there is "full" mode in flow control, does he mean the buffer is full ? Or the dupelx mode ?

In GE, there are only "receive" and "desired for transmit" mode. The reason of display desired, it was because you configured it as desired receive. You can set it "on". Please check below link for the information of GE 802.3z flowcontrol :

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_guide_chapter09186a00800eb762.html#31236

Try "show port flow" to show the port status.

Hope this helps.

[chris.smailes]

Thanks for your reply,

(Sorry for my late reply - I've been on hols.)

I have asked the server administrator who says that his NIC does support flow control and whereas before the change the card reported that the flow control had negotiated to 'receive', he says it has now negotiated to 'full'.

What I don't understand is that on the 6509 ports both transmit and receive flow control are administratively set to desired - the transmit is the default setting of 'desired' and the receive is explicitly configured as 'desired'. So, why has the transmit negotiated to 'on', whereas the receive has remained at desired ? I.e. why is the behaviour different. Am I misunderstanding the negotiation process ?

The NIC does support etherchannel.

The default receive flow control setting for the port-channel of 'desired' seems to indicate that the etherchannel does support flow control and I am wondering if I need to configure this at the port-channel level to get it to work at the ethernet port level.

Many thanks for your time.

[jackyoung]

Thanks for the detail information. I am sorry that I have no idea of the reason of why 6509 to have one is desired and another not. You may try to set all to "on" to determine the result.

Moreover, for the etherchannel, it is fine to try if you have more than one NIC to group into the same etherchannel. Otherwise, if only one NIC, it cannot be an etherchannel. But it is no harm to try if you have the maintenance window. Hope I can find more info. for this situation.

What I suggest is configure all parameters manually (speed, duplex, flowcontrol) then try again.

Hope this helps.

[JCOTSIKOROS]

Hi all, i have been looking in to flow-control for a while now and have come to the conclusion that its best turned off on server -> switch connections. Although it might not be the best option for all circumstances, if you are running just TCP/IP over that link, then TCP has its own network layer measures for managing congestion, retries, timeout etc, so why have another process that could confuse and interfere with TCP.

Try getting the admin to turn off flow-control (send and recieve) and you do the same and see if the errors continue.

If anyone has any better info on flow-control, especially any Cisco recommendations on the process, then I for one would be very interested to read them.

Thanks.

[patrick.guerin]

We saw this issue. Upon investigation the only way to prevent the issue you describe is to set flowcontrol receive and send to on for the switch ports connected to the filer. Note this can cause an outage so should be tested/configured during scheduled maintenance.

set port flowcontrol (slot/port) receive on

set port flowcontrol (slot/port) send on

or

flowcontrol send on

flowcontrol receive on

Without enabling this feature and leaving at the default state the data flowing between the switch port and the filer is constrained as the filer NIC is unable to keep up with the data that is being received at its buffers thus ensuing a queue overflow error. Enabling this feature allows for negotiation between the NetApp filer and switch port causing pause packets to be sent to other device ports to stand off the data for a short period of time.

It has been confirmed that if flowcontrol is not configured on the switch port the problem may not show up until the filer is rebooted.

Some more info from Netapp - see appliance network settings

http://www.netapp.com/library/tr/3369.pdf