Problem using multiple port groups on a WS-X6716-10GE

[spfister336]

We are trying to add a load balancer to a 10GE port on a WS-X6716-10GE on a Cisco 6509E switch. When the load balancer is online, we experience strange issues and I'm starting to suspect it may be a configuration issue on the X6716.

All port groups on the X6716 are in oversubscription mode. Currently, only port 5, in port group 2, is in use. The load balancer is being added to port 1, in port group 1. As soon as the load balancer's port is configured, the issues start, even before traffic starts flowing to it. Is there some issue we might be running into using multiple port groups?

[Jon Marshall]

That module has 2 x 20Gbps connections to the switch fabric.

Port groups 1 and 2 share one of the connections and port groups 3 and 4 share the other.

So how many ports from groups 1 and 2 are in use ?

Jon

[spfister336]

Port 5 in group 2 is the only thing currently in use.

Port 1 in group 1 is where the load balancer would be added.

Is it 2 x 20Gbps? I thought it was 4 x 10 Gbps...  either way, there isn't nearly even 10 Gbps going through this module currently.

[Jon Marshall]

The module has 2 fabric channel connections according to the release notes and a total of 40Gbps to the switch fabric so it must be 2 x 20Gbps.

If you are only using two ports out of port groups 1 and 2 then they are not oversubscribed as long as you don't use another port out of those groups.

You say strange issues but you haven't said what they are ?

Jon

[spfister336]

Sorry... I should have been a little more specific. Unfortunately, it's been hard to find opportunaties to troubleshoot this issue. Port 2 currently leads to a stack of 2 4500X switches. When the load balancer port is active, some devices become unreachable. It seems like all of them are behind the 4500x stack. Just by disconnecting the load balancer cable, they are reachable again.

[Jon Marshall]

Difficult to say but load balancers often do NAT on IPs.

Is there any chance the load balancer is somehow responding for client IPs connected to the stack ?

How is the load balancer connected ie. is it trunked and if so does it share some of the vlans going to the stack ?

If the stack is routing for it's vlans then just ignore the above but if the routing is done on the 6500 it may be worth checking the arp cache on the 6500 when you have the load balancer connected.

I should say I doubt that is the issue but nothing else springs to mind at the moment.

Is there a pattern to the clients that are affected ie. same vlan for example ?

Jon

[spfister336]

The 6509 port is an access port in a certain vlan (99). That goes to the inside port on the load balancer. The outside ports go to two firewalls (active/standby pair). The ports on the firewall they're connected to are in that same vlan. The same vlan is used on the 4500x stack. Routing for the vlans happens on the 6509.

This load balancer is part of a new content filter solution. The way the ports are set up is exactly the way the currently used solution is set up. In addition to moving to new equipment, the port on the 6509 is going from a gig port to a 10GE port.

[Jon Marshall]

So the load balancer is in L2 mode by the sounds of it.

So where is the default gateway for vlan 99 is it the 6500 or the is the gateway on the firewalls ?

Is it just clients in vlan 99 on the stack that are affected ?

Jon

[spfister336]

The default gateway for that vlan is on the 6500. We've tried twice now to put the load balancer online, and I haven't had much time to troubleshoot. We're going to schedule a maintenance window at some point to do more investigation.

[Jon Marshall]

Is it just clients in vlan 99 that are affected ?

If so and the load balancer is L2 then also check the firewalls because they will be seeing any traffic from the clients and firewalls can sometimes respond depending on their NAT configuration.

If it just those clients then check their default gateway which should obviously be the L3 SVI on the 6500.

It sounds like something ie. the load balancer or the firewall is responding to something it shouldn't be.

Jon