Solved: service down on secondary CSS

skumar1969 · ‎03-22-2005

Hi,

I have a pair of CSS providing HA, one as pri and the other as backup. They have ASR between them. The configs are symmetric except the SSL keys.

On the server segment I use pair of L2 switches to provide resiliency. Etherchannel configured between them on couple of 10/100 port. Each server uses NIC-teaming on the interfaces and connects to both the L2 switches.

Each server has got port 80 and 90 in production. So there are a total of 4 services configured with L4 KAL on each CSS.

The issue is primary CSS has all the 4 services up. But on the sec I see only 2 of them active the other 2 are down. L4 connectivity using icp probe against those 2 ports(port 90) fetches nothing. Actually these ports are working fine with the primary. Any clue?

thanks in advance.

Gilles Dufour · ‎03-25-2005

so, you can ping it but the keepalive is down.

Actually we can even see that the server response is a 302 redirect.

What version do you run ?

There are bugs related to tcp-close fin.

CSCeg60264 - "keepalives remain in a state"

fixed in version 7.40(1.07)s

Regards,

Gilles.

View solution in original post

Gilles Dufour · ‎03-22-2005

sniffer trace.

We need to know if the server respond and with what source/destination mac address.

Teaming is usually a source of trouble.

Gilles.

skumar1969 · ‎03-25-2005

Hi Gilles,

Here is the config of both the primary and the back up CSS. The issue was that the services with similar config is up on primary but not on backup. There is a pair of L2 switches between the CSS and the server farm. We use CSS on Routing mode, 2 different VLANs one for client and another for server segment.

CSS-primary# sh run ser server1-http

!************************** SERVICE **************************

service server1-http

ip address 172.16.111.71

protocol tcp

port 85

keepalive tcp-close fin

keepalive type tcp

keepalive port 85

active

CSS-primary# sh ser summary |grep http

server1-http Alive 0 1 2 0

server2-http Alive 0 1 2 0

server3-http Alive 0 1 2 0

CSS-primary# llama

CSS-primary(debug)# icp probe service server1-http

Probing 172.16.111.71:85(-) KeepAlive probe (9)

IP Address: 172.16.111.71

Port: 85

URL: /

HTTP Version: 1.1

Server Model: Microsoft-IIS/6.0

Server Date: Fri, 25 Mar 2005 10:01:52 GMT

HEAD Response: 302 Moved Temporarily

Location: /login.aspx?ReturnUrl=%2fDefault.aspx

HEAD Support: Yes

Persistence: Yes

Keep-Alive: No

Request Depth: 14

TBR: Unknown

Connect Time: 1 ms

Rqst/Rsp Time: 3 ms

Pipeline: No

SSL: No

CSS-primary(debug)#

___________________________________________________________________

CSS-Backup# sh run ser server1-http

!************************** SERVICE **************************

service server1-http

ip address 172.16.111.71

protocol tcp

port 85

keepalive tcp-close fin

keepalive type tcp

keepalive port 85

active

CSS-Backup# sh ser summary |grep http

server1-http Down 0 1 255 0

server2-http Down 0 1 255 0

server3-http Down 0 1 255 0

CSS-Backup(debug)# icp probe service server1-http

Probing 172.16.111.71:85(\) KeepAlive probe (14)

IP Address: 172.16.111.71

Port: 85

URL: /

HTTP Version: 1.1

Server Model: Microsoft-IIS/6.0

Server Date: Fri, 25 Mar 2005 09:52:48 GMT

HEAD Response: 302 Moved Temporarily

Location: /login.aspx?ReturnUrl=%2fDefault.aspx

HEAD Support: Yes

Persistence: Yes

Keep-Alive: No

Request Depth: 14

TBR: Unknown

Connect Time: 1 ms

Rqst/Rsp Time: 2,463 ms

Pipeline: No

SSL: No

CSS-Backup(debug)#

thanks

jfoerster · ‎03-25-2005

HI,

are you able to ping server-1,2,3 from the backup CSS? Are you sure you are having proper layer 2 connectivity from the backup CSS to the servers?

Is there a trunk between the two CSSes?

How does the layer 2 look like? Could you please attach a brief drawing?

Kind Regards,

Joerg

skumar1969 · ‎03-25-2005

Hi,

CleintSide-->CSS primary--->L2SW-->Server1

|

|EtherChannel

|on Fa0/1 & Fa0/24

|

CleintSide-->CSS backup--->L2SW-->Server1

The CSS are running APP session between them on their dedicated ISC ports. L2 SW are configured to run a Etherchannel on their Fa 0/1 and Fa 0/24.

Refer my previous post and the icp probe done under the debug mode. It shows the connectivity to L4 on the server ports are OK which means connectiivyt at L3 & L2 levels should be OK.

thanks

Gilles Dufour · ‎03-25-2005

Your service is defined as L4 and it shows as down.

So you can't even establish a tcp connection between css and server.

So, please, verify you can ping the service from the backup CSS.

If you can't, check arp entry on CSS and server and verify there is a L2 path between css and server [cam entry exist and are correct].

Once again, a sniffer trace is important if you want to avoid losing time.

Gilles.

skumar1969 · ‎03-25-2005

Gilles,

Here is the output. It seems L2 through to L4 are ok.

I have given the outputs below from primary and backup of CSS and L2 sw as well. Sorry I can not do the sniffer trace at the moment as I am sitting at remote to the CSS.

L2-pri#sh mac-address-table

Mac Address Table

-------------------------------------------

Vlan Mac Address Type Ports

---- ----------- -------- -----

100 000e.7fec.6d85 DYNAMIC Fa0/5

100 000e.7fec.6d87 DYNAMIC Fa0/6

100 0002.a54b.074e DYNAMIC Fa0/7

CSS-pri# sh arp

ARP Resolution Table:

IP Address MAC Address Type Port

172.16.111.71 00-0e-7f-ec-6d-85 dynamic 2/2

172.16.111.72 00-0e-7f-ec-6d-87 dynamic 2/2

172.16.111.73 00-02-a5-4b-07-4e dynamic 2/2

CSS-pri# ping 172.16.111.71

Pinging 172.16.111.71 1 time(s)...

Working(-) 1/1

100% Success.

CSS-pri# ping 172.16.111.72

Pinging 172.16.111.72 1 time(s)...

Working(-) 1/1

100% Success.

CSS-pri# ping 172.16.111.73

Pinging 172.16.111.73 1 time(s)...

Working(-) 1/1

100% Success.

__________________________________________________

L2-backup#sh mac-address-table

Mac Address Table

-------------------------------------------

Vlan Mac Address Type Ports

---- ----------- -------- -----

100 000e.7fec.6d85 DYNAMIC Po1

100 000e.7fec.6d87 DYNAMIC Po1

100 0002.a54b.074e DYNAMIC Po1

CSS-Backup# sh arp

ARP Resolution Table:

IP Address MAC Address Type Port

172.16.111.71 00-0e-7f-ec-6d-85 dynamic 2/2

172.16.111.72 00-0e-7f-ec-6d-87 dynamic 2/2

172.16.111.73 00-02-a5-4b-07-4e dynamic 2/2

CSS-Backup# ping 172.16.111.71

Pinging 172.16.111.71 1 time(s)...

Working(-) 1/1

100% Success.

CSS-Backup# ping 172.16.111.72

Pinging 172.16.111.72 1 time(s)...

Working(-) 1/1

100% Success.

CSS-Backup# ping 172.16.111.73

Pinging 172.16.111.73 1 time(s)...

Working(-) 1/1

100% Success.

Gilles Dufour · ‎03-25-2005

so, you can ping it but the keepalive is down.

Actually we can even see that the server response is a 302 redirect.

What version do you run ?

There are bugs related to tcp-close fin.

CSCeg60264 - "keepalives remain in a state"

fixed in version 7.40(1.07)s

Regards,

Gilles.

skumar1969 · ‎03-25-2005

Gilles,

CSS version is

sg0730307s (07.30.3.07s)

My question would be it could be a bug that is causing the trouble as you said. But why it is not affecting the primary CSS if its related to a TCP fin? Primary CSS is happy with all the services up but not the backup CSS. They both are identical twins with same hardware/image/code version etc.

thanks

Gilles Dufour · ‎03-26-2005

definitely looks like the bug I have identified.

I would suggest to simply reboot the backup and see if the services come alive.

If they do, then this is the bug and you should plan for an upgrade.

Regards,

Gilles.

skumar1969 · ‎03-26-2005

Gilles,

where would in Cisco.com I can find more about the bug you identiied?

thanks

skumar1969 · ‎03-27-2005

Hi Gilles!

The new image is a breeze and services were up. Situation under control now....thanks a bunch.