standby cisco ACE loadbalancer issues (network connectivity)

nvkiran_k · ‎02-17-2010

Hi ALL,

We are having issues with the secondary (standby) load balancer ACE module on a 6500 switch. We see that the loadblanacer is not able to get onto the network which leads to problem with fault tolerance as well. Following is the ft status found on the load balancer for one of the contexts (this is the same pattern seen on all the contexts).

switch/Admin# sh ft group status

FT Group : 1

Configured Status : in-service

Maintenance mode : MAINT_MODE_OFF

My State : FSM_FT_STATE_ACTIVE

Peer State : FSM_FT_STATE_UNKNOWN

Peer Id : 1

No. of Contexts : 1

Sh arp on all the contexts shows the gateway/rserver to be unreachable. Please find the screenshot below for one of the contexts (the same pattern is seen on the LB for all other contexts)

switch/1_Context# sh arp

Context CSD_Context

================================================================================

IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status

================================================================================

172.21.128.97 00.00.00.00.00.00 vlan942 GATEWAY - dn

172.21.128.103 00.0b.fc.fe.1b.09 vlan942 ALIAS LOCAL _ up

172.21.128.105 00.12.43.dc.93.23 vlan942 INTERFACE LOCAL _ up

7.0.0.4 00.0b.fc.fe.1b.09 vlan943 NAT LOCAL _ up

- 7.0.0.6

172.21.147.196 00.0b.fc.fe.1b.09 vlan943 ALIAS LOCAL _ up

172.21.147.198 00.12.43.dc.93.24 vlan943 INTERFACE LOCAL _ up

172.21.147.200 00.00.00.00.00.00 vlan943 RSERVER - * 3 req dn

172.21.147.202 00.00.00.00.00.00 vlan943 RSERVER - * 2 req dn

172.21.147.204 00.00.00.00.00.00 vlan943 RSERVER - dn

172.21.147.206 00.00.00.00.00.00 vlan943 RSERVER - dn

172.21.147.208 00.00.00.00.00.00 vlan943 RSERVER - * 3 req dn

172.21.147.210 00.00.00.00.00.00 vlan943 RSERVER - * 2 req dn

172.21.147.212 00.00.00.00.00.00 vlan943 RSERVER - * 1 req dn

172.21.147.214 00.00.00.00.00.00 vlan943 RSERVER - * 1 req dn

172.21.147.216 00.00.00.00.00.00 vlan943 RSERVER - * 3 req dn

7.0.0.1 00.0b.fc.fe.1b.09 vlan943 NAT LOCAL _ up

- 7.0.0.3

The problem is that we see the problem only on the secondary loadbalancer. primary is just running file

also i can see some traffic denial in admin context for resource usage

switch/Admin# sh resource usage
                                                     Allocation
        Resource         Current       Peak        Min        Max       Denied
-------------------------------------------------------------------------------
Context: Admin
conc-connections              9          9     160000    6560000          0
mgmt-connections              0         46       2000      82000          0
proxy-connections             0          4      20972     859830          0
xlates                        0          0      20972     859830          0
bandwidth                     0   17715713   10000000 535000000    5799749
    throughput                  0   17710993   10000000 410000000    5799749
    mgmt-traffic rate           0       4720          0 125000000          0
connection rate               0         43      20000     820000          0
ssl-connections rate          0          0        100       4100          0
mac-miss rate                 0          1         40       1640          0
inspect-conn rate             0          0        120       4920          0
acl-memory                56336      56336    1570072   64460552          6
sticky                        0          0      83886          0          0
regexp                        0          0      20972     859832          0
syslog buffer             82944      82944      82944    3447808          0
syslog rate                   0         44       2000      82000         25

Context: INTEGRATION_Context
conc-connections              0       3934     160000          0          0
mgmt-connections              0         98       2000          0          0
proxy-connections             0         33      20972          0          0
xlates                        0          0      20972          0          0
bandwidth                     0   10019910   10000000 125000000      40857
    throughput                  0   10000000   10000000          0      40857
    mgmt-traffic rate           0      19910          0 125000000          0
connection rate               0         49      20000          0          0
ssl-connections rate          0          0        100          0          0
mac-miss rate                 0         32         40          0          0
inspect-conn rate             0         58        120          0          0
acl-memory                11920      11920    1570072          0          0
sticky                        0          1      83886          0          0
regexp                        0          0      20972          0          0
syslog buffer                 0      82944      82944    3447808          0
syslog rate                   0        312       2000          0          0

these above 2 contexts are the only one which has bandwidth resource usage exceeding the limit. but i somehow am not sure if this is the issue. as there is just no traffic on the secondary .. then how can the bandwidth reach the threshold? can anyone throw some light on the below issue?

thanks and regards

kiran

dario.didio · ‎03-04-2010

Hi,

Can you confirm that the VLANs are passed to the ACE module using the "svclc ...." commands on the supervisor?

You should be able to see them when doing a show vlan in the Admin context.

Can you also confirm that the FT VLAN spans between the two Catalyst chassis?

HTH,

Dario

nvkiran_k · ‎03-15-2010

----------------
vlan on Standby_ACE switch

svclc multiple-vlan-interfaces
svclc module 1 vlan-group 1,4,12,13,
svclc vlan-group 1 968
svclc vlan-group 12 132
svclc vlan-group 13 367-372,374,375,379,380,538,805,807,808,818,913,915
svclc vlan-group 13 917-920,922-924,933,934,937,938,942-949,972,976-979,983
svclc vlan-group 13 984
ip subnet-zero
no ip source-route

----------------

vlans on standby ACE

switch/Admin# sh vlans
Vlans configured on SUP for this module
vlan132 vlan360 vlan367-375 vlan379-380 vlan538 vlan805 vlan807-808 vlan818 vlan913 vlan91
5 vlan917-920 vlan922-924 vlan930 vlan933-934 vlan937-938 vlan942-949 vlan968 vlan971-972 v
lan976-979 vlan983-984
switch/Admin#

----------------

Active_LB_host_switch is the switch hosting the active ACE thats connected on ten7/4 and 8/4 which is bundeled and made into
port-channel (po72)

CDP neighbor hosting the active ACE
----------------
Active_LB_host_switch
Ten 7/4 148 R S I WS-C6513 Ten 7/4
Active_LB_host_switch
Ten 8/4 156 R S I WS-C6513 Ten 8/4

Po72 allows all the vlans which is the configured for ACE modules.

Port Vlans allowed on trunk

Po72 132,140,181,359-383,538,668,702,805-808,815-816,818-820,836,907,909-920,922-925,
929-935,937-949,967-973,976-984,987,3212

vlan 968 is the FT vlan and the same hass been allowed on the trunk port.
everything looks good to me but still not sure why isnt the ACE module not coming to the network. it was working fine
a few months back but all of a sudden it lost the network connectivity. i am not even able to ping the physical ip of the
ACE module.

thanks and regards

kiran

seanbowles · ‎12-29-2010

I had a problem with the FT blades (Active/Standby) across data centers, where the FT configuration was correct, however they did not synch correctly. I had FT flapping, even without the preempt enabled. Basically, after I checked the licensing, re-installed the licenses on both modules and upgraded to the latest code, my FT problems went away. Viola! HTH, -S