FWSM ACL/NAT Issue

[gignet200]

We recently deployed a FWSM on our 6503-e boxes (w/ sup720).  NAT is working (PAT) but the issue I am seeing is private traffic from remote sites is not being allowed through the FW.   I was able to get the remote site to ping the FWSM itself (inside address), but no hosts behind it.  Maybe an ACL issue? Also when I turn off NAT on the remote end, I can than access everything (We are NATng on both ends).   Im a routing guy by nature so I will defer this to the security guys out there.   Thanks in advance.

Topology

Hosts (inside/10.15.25.0/24) > FWSM  (outside/public IP) -> Core Router -> MPLS CLOUD -> Core Router (NATng) - > Hosts (192.168.1.0/24)

ACLs applied to inside/outside interface

FWSM# show access-list ATX-ALLOW-IN

access-list ATX-ALLOW-IN; 15 elements

access-list ATX-ALLOW-IN extended permit tcp any any (hitcnt=222)

access-list ATX-ALLOW-IN extended permit icmp any any (hitcnt=101)

access-list ATX-ALLOW-IN extended permit udp any any (hitcnt=6)

access-list ATX-ALLOW-IN extended permit ip any any (hitcnt=0)

access-list ATX-ALLOW-IN extended permit tcp any any eq www (hitcnt=0)

access-list ATX-ALLOW-IN extended permit tcp any any eq https (hitcnt=0)

access-list ATX-ALLOW-IN extended permit ip any 192.168.1.0 255.255.255.0 (hitcnt=0)

access-list ATX-ALLOW-IN extended permit icmp any 192.168.1.0 255.255.255.0 (hitcnt=0)

access-list ATX-ALLOW-IN extended permit tcp any 192.168.1.0 255.255.255.0 (hitcnt=0)

FWSM# show access-group

access-group ATX-ALLOW-IN in interface outside

access-group ATX-ALLOW-IN out interface outside

access-group ATX-ALLOW-IN in interface inside

access-group ATX-ALLOW-IN out interface inside

Ping Tests

FWSM Inside address (10.15.25.245)

Host behind the FWSM (10.15.25.89)

Remote Router Inside address (192.168.1.1)

FWSM to remote spoke site Router

FWSM# ping 192.168.1.1

    192.168.1.1 response received -- 10ms

    192.168.1.1 response received -- 20ms

    192.168.1.1 response received -- 10ms

Remote Router to FWSM

ATX-CFW1#ping 10.15.25.245

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.15.25.245, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms

Remote Router to a host behind the FWSM

ATX-CFW1#ping 10.15.25.89

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.15.25.89, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

[Jouni Forss]

Hi,

I'm not 100% sure on your setup.

Are the networks 10.15.25.0/24 and 192.168.1.0/24 LAN networks on each site?

Because to my understanding you should only be able to PING an FWSM (or any Cisco firewall product) interface IP address from behind that said interface. And looking at the above post seems to point out that looking from the remote site the FWSM interface IP address is not the IP address facing the remote site but the LAN side of the FWSM?

You say also that you are NATing both ends. Are the above networks NATed networks or the real IP addresses?

Also on the FWSM side I dont see a benefit to having the same ACL attached to so many directions and to several interfaces. The most typical case is to use a separate ACL for each interface and only use them in the direction "in" controlling traffic entering each interface. You dont have to take into account the return traffic with a firewall.

Can you clarify the setup abit with regards to the networks and their actual location related to the FWSM and remote router.

- Jouni

[gignet200]

Thanks for the quick response.  10.15.25.0/24 is the LAN side behind the FWSM, and 192.168.1.0/24 is the LAN side behind the remote router.  So what is happening is we are natting the 10.15.25.0/24 network to a public IP (outside interface of FWSM), and on the remote end we are also natting the 192.168.1.0/24 address to a public IP.  I am using BGP inbteween the CORE routers to advertise both the 10.15.25.0/24 ; 192.168.1.0/24 private networks to each other.  From the remote site I can most def ping the inside IP of the FWSM, so now i just need to figure out how to ping hosts behind the 10.15.25.0/24 network.  Thanks again

[Jouni Forss]

Ok,

I can't comment much on the router side when it comes to NAT. Last time I have configured NAT on a router must have been in school Though I configure our core routers regularly.

Now on the FWSM side though IF you have configured the PAT translation for the LAN network 10.15.25.0/24 towards the remote site this would mean that if you want that LAN network to reach some remote network with its original IP address and also be visible to it with the original IP address from the remote sites perspective, you would have to configure NAT0 that applies to the connections between the LAN networks of each site.

The reason for the above is that all traffic from the FWSM LAN should be hitting to the PAT rule when the direction is FWSM LAN -> Any other network behind the "outside"

To configure a simple NAT0 / NAT Exempt that applies when the source network is 10.15.25.0/24 and the destination network is 192.168.1.0/24 the configuration would be the following.

access-list INSIDE-NAT0 remark NAT0 for FWSM LAN and Remote LAN

access-list INSIDE-NAT0 permit ip 10.15.25.0 255.255.255.0 192.168.1.0 255.255.255.0

nat (inside) 0 access-list INSIDE-NAT0

This should enable the traffic between the 2 LANs (with regards to the FWSM configuration, I cant comment on the router side). Do take into account that I dont know the rest of your networks configurations so I dont know if this would have any effect on something in production.

What this should simply make it so that FWSM does no NAT between these 2 LAN networks BUT rest of the traffic from FWSM LAN 10.15.25.0/24 will still get PATed on the way through the FWSM "outside" interface.

I'm still kinda wondering how it is possible to ping an FWSM interface IP address that is NOT facing the remote site. Either I am missing something or its some software related thing. But to my understanding it should not work.

- Jouni

[Jouni Forss]

Also,

Also check that you have this configuration enabled

policy-map global_policy

class inspection_default

   inspect icmp

This should enable any ICMP Echo reply message to pass through the FWSM as long as the actual Echo message has been allowed through first. Though in your case since you have allowed all traffic I guess this might not be needed. In the even your make the rules a bit more specific this might be good to have.

- Jouni

[gignet200]

Thankfully I went ahead and moved production traffic to another firewall,  so I can make changes to the FWSM without my users yelling at me.    I went ahead and applied the above rule, but still cannot ping from remote end to FWSM hosts.  Perhaps I need the same "NAT EXEMPT" rule on the remote side.  We are using a Cisco2851 on the remote end and we are NATng due to the users in that location requiring inet access.  Do you think I am able to ping the inside address remotley due to me allowing ICMP on the outside/inside address?  I am going to try and add a nat exempt rule on the remote end (cisco2851) and see if that does anything.  Thanks again Jouni!

[gignet200]

Would this be done on the FWSM side, or the 6503 side?  Looks like i dont have an option on the FWSM to configure this.

FWSM(config)# policy-map ?  

Type help or '?' for a list of available commands.

FWSM# show ver

FWSM Firewall Version 2.3(4)8

FWSM Device Manager Version 4.1(3)

[Jouni Forss]

Ouch

I have never even seen such an old version of FWSM I think it corresponds to having PIX on the 6.x software. Our FWSMs have been on 3.2 - 4.0 software which has different configuration format related to some configurations.

I guess it uses the old format configuration then. The command might simply be "fixup protocol icmp" or something similiar.

With regards to the FWSM and Remote router NAT configurations.

To my understanding if you have an existing PAT configuration configured for the LAN network and the destination interface for that PAT  (where the PAT address is located) is the interface where the remote network is also located, you will always have to exempt traffic from NAT if you want to make it possible for the said LAN network to pass through without any form of NAT. (As otherwise all traffic hits the PAT rule when passing "inside" -> "outside")

Hopefully I am making sense. I am usually halfway through a sentence when I start thinking if I have explained the situation in a way too complex way compared to how it could be explained

- Jouni

[gignet200]

  Thats the version the card was shipped with, so I guess im stuck with it for now.  I do have the "fixup protocol icmp" applied.  The fact that if I turn off NAT on the remote end and i can ping, tells me it has to be some kind of NAT translation issue.  I have added a NAT exempt rule on the remote router, but I still cant seem to ping FWSM hosts.   Can you think of anything else I might be missing?  Thanks J!

FWSM

FWSM# show access-list INSIDE-NAT0

access-list INSIDE-NAT0; 2 elements

access-list INSIDE-NAT0 remark NAT0 for FWSM LAN and Remote LAN

access-list INSIDE-NAT0 extended permit ip 10.15.25.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=10)

access-list INSIDE-NAT0 extended permit ip 10.15.25.0 255.255.255.0 host 198.205.5.82 (hitcnt=0)

FWSM# show nat

nat (inside) 0 access-list INSIDE-NAT0

Remote Router

ip nat outside source list 100 interface GigabitEthernet0/0

access-list 100 deny   ip 192.168.1.0 0.0.0.255 10.15.25.0 0.0.0.255

access-list 100 permit ip 192.168.1.0 0.0.0.255 any

access-list 100 permit ip 10.15.35.0 0.0.0.255 any

ip nat inside source list 100 interface GigabitEthernet0/0 overload

ip nat outside source list 100 interface GigabitEthernet0/0

[Jouni Forss]

Hi,

A quick look at a configuration guide for routers would make it seem like the only configurations on a very default Router NAT configuration would be

access-list 100 deny   ip 192.168.1.0 0.0.0.255 10.15.25.0 0.0.0.255

access-list 100 permit ip 192.168.1.0 0.0.0.255 any

access-list 100 permit ip 10.15.35.0 0.0.0.255 any

ip nat inside source list 100 interface GigabitEthernet0/0 overload

Which should do PAT for the 2 mentioned LAN networks behind the routers while using the interface GigabitEthernet0/0 IP address as the PAT address. The first "deny" statement should to my understanding exempt the specified traffic from NAT alltogether.

I dont know what the purpose of the configurations are that are using the "ip nat outside"

- Jouni

[gignet200]

I think once I get the nat exempt setup correctly on both ends, we will be in business.  Dont mind my ip nat outside command, I was just trying it to see if it had any impact.  I have also modified my ACL on the remote end.  I will have to wait until after 5pm central to make these changes (dont want to take down user traffic and get yelled at).  Here is the config I am planning on applying.  how does it look to you?  Thanks J.  Its nice to have someone to bounce these ideas off of.

Remote Router:

Extended IP access list NO-NAT

    10 deny ip 10.15.25.0 0.0.0.255 192.168.1.0 0.0.0.255

    20 permit ip 192.168.1.0 0.0.0.255 any

ip nat inside source list NO-NAT interface GigabitEthernet0/0 overload

[Jouni Forss · Accepted solution]

Hi,

To my understanding you router NAT configuration should have the local network of 192.168.1.0/24 as the source and the FWSM LAN network 10.15.25.0/24 as the destination network.

- Jouni

[gignet200]

Good Catch.  I have altered the ACL to look like the following.  I think this just may work.  Thoughts?  Thanks

Remote Router:

ip access-list extended NAT-RULES

deny   ip 192.168.1.0 0.0.0.255 10.15.25.0 0.0.0.255

permit ip 192.168.1.0 0.0.0.255 any

ip nat inside source list NAT-RULES interface GigabitEthernet0/0 overload

[gignet200]

After applying the NAT exempt rules on each end, I can only ping one way.  From the Remote PC (PCB) I am not able to ping anything passed the FWSM.  As a test, I put a SVI on the MSFC, and I can succesfully ping that from PCB.  That tells me its still the fwsm somehow blocking it.  I will say this is some progress at least from not being able to ping anything.  Does anything in my configs look a little off?  Thx

Topology

PCA  - FWSM - MSFC - Core1 - MPLS - Core2 - CFW - PCB

PCA: 10.15.25.89

FWSM: 10.15.25.245

MSCF: 10.15.25.57

CFW: 10.15.35.1

PCB: 10.15.35.222

PCA to PCB Ping Tests

PCA > ping 10.15.35.222

Pinging 10.15.35.222 with 32 bytes of data:

Reply from 10.15.35.222: bytes=32 time=15ms TTL=125

Reply from 10.15.35.222: bytes=32 time=14ms TTL=125

Reply from 10.15.35.222: bytes=32 time=14ms TTL=125

Reply from 10.15.35.222: bytes=32 time=14ms TTL=125

Ping statistics for 10.15.35.222:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 14ms, Maximum = 15ms, Average = 14ms

PCB to PCA/MSFC Ping Tests

PCB > ping 10.15.25.89

Pinging 10.15.25.89 with 32 bytes of data:

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Ping statistics for 10.15.25.89:

    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

PCB >ping 10.15.25.57

Pinging 10.15.25.57 with 32 bytes of data:

Reply from 10.15.25.57: bytes=32 time=15ms TTL=253

Reply from 10.15.25.57: bytes=32 time=15ms TTL=253

Reply from 10.15.25.57: bytes=32 time=14ms TTL=253

Reply from 10.15.25.57: bytes=32 time=14ms TTL=253

Ping statistics for 10.15.25.57:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 14ms, Maximum = 15ms, Average = 14ms

FWSM Config:

FWSM# sh nat 

nat (inside) 0 access-list LAN-NO-NAT

access-list LAN-NO-NAT; 2 elements

access-list LAN-NO-NAT extended permit ip 10.15.25.0 255.255.255.0 10.15.35.0 255.255.255.0 (hitcnt=120)

access-list LAN-NO-NAT extended permit ip host X.X.X.X 10.15.35.0 255.255.255.0 (hitcnt=0)

X.X.X.X = PUBLIC IP OF REMOTE ROUTER

CFW Config

Extended IP access list nat_source

    10 deny ip any 10.15.25.0 0.0.0.255 (6304597 matches)

    20 deny ip any host 198.205.5.66

    30 permit ip 10.15.35.0 0.0.0.255 any (455 matches)

ip nat inside source list nat_source interface GigabitEthernet0/0 overload

Routing tables on both FWSM and Core2

FWSM

FWSM# show route

O    10.15.35.0 255.255.255.0 [110/12] via 198.205.5.65, 0:51:40, outside

CFW

#show ip route 10.15.25.0

Routing entry for 10.15.25.0/24

  Known via "ospf 500", distance 110, metric 3, type intra area

[Jouni Forss]

Hi,

Are we talking about a FWSM in "mode transparent" ?

- Jouni