Missing route behaviour

[gabriel.gearip]

Hi all,

I had a problem yesterday which I cannot explain; the setup is similar to the one in the attachment.

Networks 172.16.1.0/24 and 172.17.1.0/24 are routed on the two routers towards the outside interface of the firewall. The network 172.16.1.0/24 has a route towards the LAN (routed on a downstream L3 switch) but 172.17.1.0 has no route on the routing table of the firewall.

Host 10.1.1.1 was sending constant traffic to network 172.16.1.0/24. At one point it started sending traffic also to network 172.17.1.0/24; this led to an overload of the firewall with some 10000 duplicate SYN packets/second resulting in high CPU load on the firewall and an disruption of other legitimate traffic. Here's a sample from the syslog:

Apr  7 10:03:59 192.168.1.1 %ASA-4-419002: Duplicate TCP SYN from outside:10.1.1.1/4638 to outside:172.17.1.16/1617 with different initial sequence number

Apr  7 10:03:59 192.168.1.1 %ASA-4-419002: Duplicate TCP SYN from outside:10.1.1.1/61061 to outside:172.17.1.16/2156 with different initial sequence number

Apr  7 10:03:59 192.168.1.1 %ASA-4-419002: Duplicate TCP SYN from outside:10.1.1.1/39075 to outside:172.17.1.16/229 with different initial sequence number

I solved the problem by routing network 172.17.1.0/24 to the inside interface of the firewall, thus sending the traffic to the LAN. The avalanche of duplicate TCP SYN packets stopped.

Still I cannot explain the behaviour of the firewall.

Do you have an ideea?

Thanks.

[IAN WHITMORE]

Without a lot more information (like configs etc) I can only think of the following...

The traffic arrived on the outside interface of the firewall and with no route to the LAN it was sending it to its default route (which is the outside interface normally on a firewall). Therefore it was either in a loop with itself or in a loop with the HSRP routers??

Like I say, without more info and without a lab to try and reproduce the error it is hard to tell.

I know it sounds like a syn attack, but I guess you know who the user is and I doubt they are attacking you or spoofing, right?

See: http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a00809763ea.shtml

So, sounds like a config issue on the firewall itself.

HTH,

Ian

[gabriel.gearip]

Hi Ian,

Thanks for the answer, you're probably right about the routing loop with the HSRP routers.

But I'd like to discuss it more... The remote host (10.1.1.1) is sending rather low amounts of traffic, a few packets per minute. Taking into account the TTL of the packets and even if the host was sending 60 packets per second to the route-less network I shoulnd't have seen more than 120 SYN packets per second on the firewall interface. But I was seeing an avalanche of about 10000 packets/second; could the packets be multiplying? How can this be explained?

Thanks again!

[IAN WHITMORE]

Hmmm...what routing protocol are you using? What is the max. hop count? Remeber that the routing decisions are sub second so 1 packet could be looped many times in one second. Of course you would need a steady flow of packets from the host, but once in the loop you would see loads of entries in your log per second - using your example of 60 pps I would expect thousands of entries in the firewall log per second until the packets reach max. hop count and are dropped.

Woudln't be great to recreate this in a lab environment?

[gabriel.gearip]

Sorry, I mistyped, I meant 60 packets per minute (so 1 per second)...

BGP is used in the WAN (right of the HSRP routers), static routing towards the LAN. In the LAN OSPF is used but this has no bearing here as the firewall has only a default route toward the WAN.

I will actually recreate the scenario in the lab tommorow with the original config of the firewall and let you know the results . Unfortunally I don't have the configs of the HSRP routers as they're out of my management.

[gabriel.gearip]

I reproduced the case in the lab using a firewall and a routers. I got 126 SYN packets for every connection, so as expected.

I extracted from the log files the duplicate SYN connections and got an average of 3000 packets/second, so that would mean 23 original connections/second. Pretty high...