Can I get closer to 2gbps on a 2 port port-channel to a backup appliance?

[keithsauer507]

Hello, configured an Exagrid backup appliance with two nics in bond0 and appliance states 802.3ad is the mode.

These two links are connected to a 3560-X switch and Port-channel 12 was defined.

I ran a backup job and monitoring the traffic graphs I see that one of the nic's is barely used, while another one is carrying the bulk of the traffic (500mbps+).  Also doing a bps command on the backup server only seems to indicate one nic is receiving the backup traffic.

[root]# bps 2
eth2: 130,820 bytes TX/sec; 24 bytes RX/sec
eth2: 143,338 bytes TX/sec; 0 bytes RX/sec
eth2: 136,926 bytes TX/sec; 0 bytes RX/sec
eth2: 145,504 bytes TX/sec; 0 bytes RX/sec
^C
[root]# bps 3
eth3: 200,142 bytes TX/sec; 84,126,652 bytes RX/sec
eth3: 181,807 bytes TX/sec; 73,265,194 bytes RX/sec
eth3: 183,760 bytes TX/sec; 83,121,268 bytes RX/sec
eth3: 154,606 bytes TX/sec; 59,581,553 bytes RX/sec

The switchport configuration is as follows:

interface Port-channel12
description Exagrid Backup System
switchport access vlan 10
switchport mode access

interface GigabitEthernet1/1
description Exagrid Backup System nic3
switchport access vlan 10
switchport mode access
channel-group 12 mode active
!
interface GigabitEthernet1/2
description Exagrid Backup System nic4
switchport access vlan 10
switchport mode access
channel-group 12 mode active

A show of the int Po12 shows a bandwidth of 2000000 Kbit but indicates Full-duplex, 1000 Mb/s.

Port-channel12 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 4403.a7d1.7499 (bia 4403.a7d1.7499)
Description: Exagrid Backup System
MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,
reliability 255/255, txload 70/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is unknown
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/1 Gi1/2
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2628000 bits/sec, 5122 packets/sec
5 minute output rate 556221000 bits/sec, 45843 packets/sec
15182077 packets input, 972462629 bytes, 0 no buffer
Received 7263 broadcasts (5714 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5714 multicast, 0 pause input
0 input packets with dribble condition detected
135584603 packets output, 205456905904 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Is there anything I'm missing here or is this just the nature of the beast and I should check back if I have multiple jobs running that could potentially see in excess of the current traffic rate?  I just want to make sure I have this connection optimal.

[Jon Marshall]

For a specific flow etherchannel will only use one of it's links ie. it cannot spread the same flow across multiple links in the etherchannel.

How a flow is defined is dependant on the devices themselves in terms of how they can distinguish between flows eg. src/dst mac address, src/dst IP address, src/dst ports etc.

So if the backup is from the one server to the backup server using the same ports etc. then etherchannel will only use one of the links.

If however you then started another backup from another server for example there is a good chance the switch would use a different link.

Jon

[keithsauer507]

If however you then started another backup from another server for example there is a good chance the switch would use a different link.

Ok this is what I suspect.  Right now we are only running one backup during the day as a test, to this new backup appliance.  After hours a whole slew of backups will run.  I'll be able to check our bandwidth graphs tomorrow and see how the link handles a larger traffic flow.

I did just want to make sure that the port channel configuration was correct above.  I've always used that port channel configuration between other switches or servers with bonded nics.

[Jon Marshall]

The configuration is fine but the config does not show the load balancing method.

Should probably explain a little more about how it works.

Lets say all the servers are on a remote IP subnet from the backup server and you load balance based on src and dst mac address.

That would be a bad thing to do because the src and dst mac addresses will always be the same ie. the mac address of the backup server and the mac address of the backup servers default gateway.

So you would want to load balance on src/dst IP or a combination assuming your switch supports it ie. you need to make sure it is using the most appropriate algorithm for your traffic which it may be by default.

Note also not all switches support all algorithms.

Have a look at this link which goes into it in a lot more detail and if you have more questions then come back -

http://www.cisco.com/c/en/us/support/docs/lan-switching/etherchannel/12023-4.html

Jon

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I agree with Jon, generally a good hash setting for Etherchannel, for devices that support it, is src/dst IP (which the 3560-X, I recall, does support, but I also recall, is not its default.)

BTW, even with "ideal" hashing, realize it's static based on frame/IP attributes, it is not dynamic load based, so two busy flows can be directed to the same link while another link is sitting idle.

[keithsauer507]

Just out of curiosity, if one were to change the has mode to src/dst IP, would one have to also do this on every single upstream switch that this one is connected to (with portchannels)?

This also has a port-channel upstream to a 3750x2 stack.  That 3750x2 stack has multiple port channels.  So if changing one means changing all, that could be a very daunting process and likely require an entire network reboot.

[Jon Marshall]

No the algorithm does not need to be the same on all etherchannels.

In fact you can have a different algorithm in use on either end of the same etherchannel if that is what you need.

Jon

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

As noted by Jon, Etherchannel hash doesn't have to be the same across all devices, and in some cases, shouldn't be.  Further, as individual device model series have different defaults, some series might already use the hash algorithm that's best for that particular device.

Of course, for any individual device, if the active hash algorithm isn't ideal, you'll likely want to change it, and yes, that can be a bit of work in a large network using lots of Etherchannel.

Some good news, though, I believe most if not all Cisco devices don't need to be rebooted to activate a different hash algorithm.  However, some in-flight traffic might be impacted, so best to make the change during a maintenance window.

[Karsten Iwen]

BTW, even with "ideal" hashing, realize it's static based on frame/IP attributes, it is not dynamic load based, so two busy flows can be directed to the same link while another link is sitting idle.

A little anecdote from some years ago:

The client complained that all the etherchannels would not work and we didn't implement it right. They just implemented a network-monitoring and the interface utilization showed that on all etherchannels only one of the two members was active.

It took us some time to realize what the problem was: For whatever reasons, all servers and nearly all clients (no DHCP) were only using even IP addresses and the load-sharing algorithm calculated the same link-member for all traffic.