Keepalives coming from external IPs?

[brian]

We've been having some strange behavior around 6:30 in the morning every weekday lately for our 3 LB'd server behind our CSS 11151 (v 5.00 Build 610s) with lots of up/downs on our keep alives

Our network config is a pretty standard (a relative term) two-armed setup.

Keepalives are one global per server GET'ing /keepalive.txt

Services reference the named KA for that box.

In the course of investigating this, I was looking at the Apache logs on one of the servers and found this strange set of entries:

24.19.12.47 - - [16/Aug/2006:06:31:49 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

24.19.12.47 - - [16/Aug/2006:06:31:50 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

72.23.85.21 - - [16/Aug/2006:06:31:50 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

(same line repeated 9 times)

67.189.136.30 - - [16/Aug/2006:06:31:50 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

(same line repeated 20 times)

10.0.15.1 - - [16/Aug/2006:06:31:50 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

24.19.12.47 - - [16/Aug/2006:06:31:50 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

72.23.85.21 - - [16/Aug/2006:06:31:50 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

24.19.12.47 - - [16/Aug/2006:06:31:50 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

10.0.15.1 - - [16/Aug/2006:06:31:54 -0500] "GET /keepalive.txt HTTP/1.0" 200 7 "-" "Mozilla/4.06 [en] (WinNT; I)" 10.0.15.12

The only IP in there that should be the source of a keepalive request is the 10.0.15.1 address, the ip of the CSS. No one else knows about that keepalive file.

Now, I easily found 24.19.12.47 in the logs doing normal sorts of requests right before and after this block of entries.

This seems like highly anomalous behavior on the part of the CSS.

I found similar entries in the logs of all 3 web servers.

Any ideas as to why this is happening?

Is this a cause or an effect of our drops in availability?

TIA

Brian

[r.docuyanan]

Hi Brian,

Have you enabled security on the CSS?, Do you have SNMP enabled to log DOS attacks?

[brian]

1. Yes.

2. I did not see anything the logs for the CSS stating it thought an attack was underway.

3. I'm pretty sure it's not a DOS attack as there is actually a DROP in traffic to the system during this time period.

[Gilles Dufour]

Brian,

is this an ip of the CSS itself ?

If not, I don't see how the css could generate keepalive using an ip address that does not belong to itself.

You should maybe capture sniffer traces to see what is going on.

Capture front-end and back-end traces.

Gilles.

[brian]

Yes, 10.0.15.1 is the CSS's address on that VLAN.

The 10.0.15.12 address seen in the Apache log is the IP address of the web server. The CSS is making this call for "web2":

http://10.0.15.12/keepalive.txt

As for captures, I will do that tomorrow morning.