GLBP with PBR

[johnnylingo]

So I've configured GLBP on a pair of Cisco 6509 switches.  It should be balancing 50/50, but instead I see 1/99.  Any ideas what the problem could be?  There's a pre-existing Policy-Based routing configuration, and I'm wondering if that would interfere with GLBP operation.  

R1 is the STP root, yet R2 is taking 99% of the input traffic.  

R1

interface Vlan1
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 no ip proxy-arp
 ip route-cache policy
 ip policy route-map PBR_ROUTE-MAP
 load-interval 30
 glbp 1 ip 192.168.1.1
 glbp 1 priority 255
 glbp 1 preempt
 glbp 1 weighting 254
 glbp 1 load-balancing weighted

R2

interface Vlan1
 ip address 192.168.1.3 255.255.255.0
 no ip redirects
 no ip proxy-arp
 ip route-cache policy
 ip policy route-map PBR_ROUTE-MAP
 load-interval 30
 glbp 1 ip 192.168.1.1
 glbp 1 priority 254
 glbp 1 preempt
 glbp 1 weighting 254
 glbp 1 load-balancing weighted

[Jon Marshall]

How are you measuring the amount of traffic ?

When you say input do you mean from clients ?

PBR should only affect traffic once it has arrived at the interface ie. it should not affect which 6500 the client uses to send traffic to.

Jon

[johnnylingo]

I'm just looking at input rate on the SVIs.  R1 is doing 114 Mbps while R2 is doing 3.8 Gbps.

R1>sh int vl1
Vlan1 is up, line protocol is up 
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 3/255, rxload 29/255
  30 second input rate 114621000 bits/sec, 10089 packets/sec
  30 second output rate 14325000 bits/sec, 23138 packets/sec

R2>sh int vl1
Vlan1 is up, line protocol is up 
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 130/255
  30 second input rate 3807281000 bits/sec, 337836 packets/sec
  30 second output rate 5921000 bits/sec, 10625 packets/sec

[Jon Marshall]

Couple of things to bear in mind -

1) traffic from the other "side" ie. return traffic to the client vlan can use either switch

2) the rates do not tell you how much traffic individual clients are sending or receiving ie. you may have some very busy clients all tied to the same switch

Not saying you are not load balancing but a better way to check would be to look at the clients and see what mac address they are using for their default gateway.

You don't need to look at all clients, after all if all clients are tied to the same switch you should be able to see that pretty quickly.

Jon

[johnnylingo]

Yes, the return traffic is controlled by Multipath BGP, so that's a different beast.  I'm strictly trying to figure out why R2 is taking so much input traffic. Even if I set the GLBP weight on R1 to 254 and R2 to 1, it has no effect.  And yes, I did wait several hours for ARP to be refreshed.

There are about 200 clients on the VLAN, 50 of which are doing the heavy traffic.  So while I don't expect the traffic to be a perfect 50/50, I would expect to see a better result than 1/99.  The "show glbp" command shows that both switches are sending about the same number of ARP replies.  

[Jon Marshall]

It's difficult to say from just the SVI stats.

If 50 clients are doing heavy traffic and the majority of these happened to end up with the same switch as their gateway then you would expect to see this.

You really would have to look at the end clients and see what they think is their default gateway in terms of mac address because each switch will be handing out a different one.

Jon

[johnnylingo]

This looks like it's an issue with the clients.  When I logged in to them, all were resolving 192.168.1.1 to 00:07:B4:02:5C:01.  I deleted the ARP entry manually, and at that point they refreshed to 00:07:B4:02:5C:01 half the time and 00:07:B4:02:5C:02 the other half.  

The clients are Linux servers, so they should be refreshing every 60 seconds. Apparently there's behavioral a difference between an ARP entry timing out and it being manually deleted.