Cisco FWSM NAT Table Issue

[John Apricena]

Hello All,

We had an issue today with one of our servers that sits behind our FWSM. This server has a static NAT entry with a private and public IP Address and ports opened to the outside world. It is one of a few email servers and two of these ports that are opened are 587 and 443. For some reason connections from the outside to this server were failing today, and after troubleshooting, I noticed that when I would try to connect on those ports, I wouldn't even see the connections appear in the logs for only that IP. All other IP's behind the Firewall from the outside worked fine. I cleared the xlate for this IP, and it resolved. Do any of you guys know how or why this could have happened, as I would like to avoid having to do this in the future?

Thanks in advance!!!

[Jouni Forss]

Hi,

In a static environment, meaning an environment where there is hardly any changes to the configuration, I would imagine that this is very strange.

Naturally at this point I can only guess what could have been the problem since we cant take any outputs from the devices in question.

If you didnt see any connections come to your firewall at all for this public NAT IP address in question then it seems that it might be likely that the upstream router wasnt perhaps able to determine the MAC address of the public IP address used for NAT. And by that I mean that the firewall might have stopped answering to ARP request.

If this was indeed the problem, I cant really say. It should be confirmed if it happened again. But if you arent seeing any connections towards the public IP address used in your NAT configuration then it would point to a problem between the FWSM and the upstream router or perhaps even some routing problem depending on your setup.

As long as the FWSM has the Static NAT configured it should to my understanding reply to ARP requests from the upstream router. That is ofcourse, if the IP address used is even part of the directly connected network configured on the FWSM "outside" interface (or whatever interface is used)

A traffic capture might naturally tell during the problem could also tell us if there truly is no traffic arriving on the firewall. Some ARP debug command might also give clues to what is happening.

- Jouni

[John Apricena]

Hey Jouni,

Thanks as always for your response. I do see the hit count on the access-list increase on the FWSM as I try to coonect to that server over ports that are open from the outside. Also, I was in the asdm and veiwing the live logs and noticed if I connect to the server on ports that aren't allowed, I get an access-list denied message on the ASDM, so thisleads me to believe the traffic is in fact getting there, but for some reason it's just not connecting. This issue happened again (almost exactly 24 hours later) this morning. We have many virtual contexts and many hosts behind each, yet this is the only host that the problem is occuring on. Since, I am seeing hit counts on the access-list for the FWSM, is it safe to rule out the upstream router or no?

[Jouni Forss · Accepted solution]

Hi,

If you are seing inbound traffic in the Security Context that holds the IP address that is having the problems WHEN the problem situation is happening then it would seem that there would be no problem with the upstream device. If we had somekind of overlap in Static NAT configirutions in other Security Contexts or perhaps somekind of overlapping route configuration then you should see constant connectivity problem.

Could it be possible that the actual servers resources arent maxed out during the problem situation? Perhaps its running some scheduled backup while its also running low on CPU/Memory resources?

Have you checked what the log outputs say for the connection attempts when you see the "Teardown TCP connection" messages? Does it say SYN Timeout? If so, then it would indicate that the server isnt answering OR that there is some other problem relaying the return traffic back to the FWSM.

You can check the connection state on the FWSM while attempting connection with the command

show conn | inc

This should list the TCP connection and its "flags". The TCP flags might tell us what is happening with the connection attempts.

There is also the option to configure traffic capture on the FWSM. I think the configurations were a bit more limited on the FWSM than on the ASA.

But to my understanding this configuration is possible. The ACL that defines the captured traffic should be specified to only include the source IP address and destination IP address of the connection so we dont get too much useless data to the capture which would make the buffer fill too fast.

access-list SERVER-CAP permit ip host host

access-list SERVER-CAP permit ip host host

capture SERVER-CAP type raw-data access-list SERVER-CAP interface buffer 10000000

The interface used should be the one that is local to the server. So we are basicly capturing traffic that is source from a public IP address BUT is destined for local IP address as the UN-NAT has already happened on the FWSM.

Then you can use the following command to view if anything is hitting the capture

show capture

You can view the actual contents of the capture on the CLI with this command

show capture SERVER-CAP

You can also copy this capture file as PCAP file to your local computer and look it through with Wireshark. This command has to be issue in the System Context of the FWSM and the "admin" Context has to have a route for the destination TFTP server and vice versa naturally

copy /pcap capture:CONTEXTNAME/SERVER-CAP tftp://x.x.x.x/SERVER-CAP.pcap

Then you can view the output with Wireshark.

What we would naturally be looking at is IF the server is sending anything back when the problem is on/happening. Naturally we are also confirming if there is any traffic coming towards the server THROUGH the FWSM Security Context.

Hope this helps

- Jouni

[John Apricena]

Thanks for this info Jouni! I will try all the above tomorroe when we expect this issue to occur again.