topic Re: Strange current connection value in Application Networking

Strange current connection value

mlanglois — Tue, 25 Sep 2007 18:36:03 GMT

We do a basic load balancing configuration for https. 3 servers. 1 is currently suspended. Today one of the servers in the content rule crashed. After it came up it showed a current 3100 connections?!?!?!? and fluctuated by a few up and down.... as time progressed.

The other server showed 20 connections. I had the server folks tell me how many active connects they had on the one that said 3000+ in the CSS and he said 3.

Any ideas? Is that some strange cosmetic bug?

Re: Strange current connection value

Gilles Dufour — Wed, 26 Sep 2007 07:37:03 GMT

what css version ?

You can do a

CSS11503-2# show flows 0.0.0.0

and see what is the list of connections.

You can also do the following procedure and see what is the timeout value for the connections

CSS11503-2# llama

CSS11503-2(debug)# flow-agent show act

-------- -------------- ----- -------------- ----- -- -------- ------- --------

Flow ID Src IP SPort Dst IP DPort Pr slot sub spt dpt Flow flg

-------- -------------- ----- -------------- ----- -- -------- ------- --------

8c5b8f60 192.168.30.112 5001 192.168.30.120 1027 6 3 1 2 1 00000528

8b71f290 192.168.30.112 1157 192.168.30.120 5001 6 1 1 2 1 00001308

CSS11503-2(debug)# flow-agent show fcb 0x8c5b8f60

Fcb Details for FCB: 0x8C5B8F60

SRC: 192.168.30.112-5001 NAT: 0.0.0.0-0

DST: 192.168.30.120-1027 NAT: 0.0.0.0-0

DMAC: 00-00-00-00-00-00 SMAC: 00-00-00-00-00-00

IP Hdr ChkD: 0 TCP/UDP Hdr ChkD: 0

TCP SequenceD: 0 Task CE: 39

BytesIn: 7064512 Frames In: 176612

Dest VLAN: 156 Src/Dst Ports: 1/0

Slot/SubSlot: 3/1 SmbQ/PrcSwP: 0/2

Time Stamp / Time Out Info:

CurSecs: 2490209:560, started: 86:441 last activity: 2490199

May timeout due to inactivity: Yes , inactiveTimeout: 16

Inactive Secs: 10, will timeout in: 6 secs

FCB Flags: 0x0528

0x0000 - Natting NOT In Use

0x0000 - NOT L5 Aware

0x0000 - Non-Spoofed

0x0008 - IP/TCP Flow

0x0000 - Local - Egress port

0x0020 - Send all to SP

0x0100 - In LL List

0x0000 - Server-side

FCB FlaFlags: 0x8040

0x0040 - Is a static FCB

0x8000 - Handled an ACK

The server may have silently dropped connections but the CSS is not timing out those connections.

Gilles.

Re: Strange current connection value

mlanglois — Wed, 26 Sep 2007 11:29:19 GMT

Version: sg0750103 (07.50.1.03)

Flash (Locked): 07.50.1.03

Flash (Operational): 07.50.1.03

Type: PRIMARY

Licensed Cmd Set(s): Standard Feature Set

But 3000 connections? I don't think this server can handle 1000 concurrent connections.. hehe. Much less 3000.

Thanks for all this info on the flows. I did a sho flow at the time and only saw about 50 flows listed. I didn't know those other commands however. You can get a lot of information about the particular flow.... but if the current local connections value showed 3000+ shouldn't I have seen 3000+ flows when I did a show flows?

Eventually.. that number just cleared itself out.

Re: Strange current connection value

Gilles Dufour — Wed, 26 Sep 2007 12:33:53 GMT

the show flow limit its output to the first 100 flows.

If the counter dropped by itself I still believe it was somehow connections for which the CSS didn't see the FIN or RESET. So they were kept alive even if on the server they had been removed. It does not mean the 3000 connections existed on the server at the same time.

Gilles.

Re: Strange current connection value

mlanglois — Wed, 26 Sep 2007 12:44:26 GMT

AH... ok.

That's good to know there is a limit...

I see... then what this may imply is that the server went down... and the CSS was still sending traffic to it? possibly?!?! Before it showed as 'dying'... and then once it was 'dead' it stopped trying to send traffic... leaving a bunch of 'half open' connections... hanging there.. ?

Is that what you are thinking?