FWSM ACL/NAT Issue

[gignet200]

We recently deployed a FWSM on our 6503-e boxes (w/ sup720).  NAT is working (PAT) but the issue I am seeing is private traffic from remote sites is not being allowed through the FW.   I was able to get the remote site to ping the FWSM itself (inside address), but no hosts behind it.  Maybe an ACL issue? Also when I turn off NAT on the remote end, I can than access everything (We are NATng on both ends).   Im a routing guy by nature so I will defer this to the security guys out there.   Thanks in advance.

Topology

Hosts (inside/10.15.25.0/24) > FWSM  (outside/public IP) -> Core Router -> MPLS CLOUD -> Core Router (NATng) - > Hosts (192.168.1.0/24)

ACLs applied to inside/outside interface

FWSM# show access-list ATX-ALLOW-IN

access-list ATX-ALLOW-IN; 15 elements

access-list ATX-ALLOW-IN extended permit tcp any any (hitcnt=222)

access-list ATX-ALLOW-IN extended permit icmp any any (hitcnt=101)

access-list ATX-ALLOW-IN extended permit udp any any (hitcnt=6)

access-list ATX-ALLOW-IN extended permit ip any any (hitcnt=0)

access-list ATX-ALLOW-IN extended permit tcp any any eq www (hitcnt=0)

access-list ATX-ALLOW-IN extended permit tcp any any eq https (hitcnt=0)

access-list ATX-ALLOW-IN extended permit ip any 192.168.1.0 255.255.255.0 (hitcnt=0)

access-list ATX-ALLOW-IN extended permit icmp any 192.168.1.0 255.255.255.0 (hitcnt=0)

access-list ATX-ALLOW-IN extended permit tcp any 192.168.1.0 255.255.255.0 (hitcnt=0)

FWSM# show access-group

access-group ATX-ALLOW-IN in interface outside

access-group ATX-ALLOW-IN out interface outside

access-group ATX-ALLOW-IN in interface inside

access-group ATX-ALLOW-IN out interface inside

Ping Tests

FWSM Inside address (10.15.25.245)

Host behind the FWSM (10.15.25.89)

Remote Router Inside address (192.168.1.1)

FWSM to remote spoke site Router

FWSM# ping 192.168.1.1

    192.168.1.1 response received -- 10ms

    192.168.1.1 response received -- 20ms

    192.168.1.1 response received -- 10ms

Remote Router to FWSM

ATX-CFW1#ping 10.15.25.245

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.15.25.245, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms

Remote Router to a host behind the FWSM

ATX-CFW1#ping 10.15.25.89

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.15.25.89, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

[gignet200]

Hey Jouni.  I set it to transparent disabled, so should be in routing mode.  Any other commands to confirm this?

FWSM#  show mode

Firewall mode: single

The flash mode is the SAME as the running mode.

[Jouni Forss]

Hi,

Im just wondering where this IP is actually located. Its part of the same network as the the segment behind the FWSM and PCA.

MSCF: 10.15.25.57

Or is it between PCA and the FWSM interface for the network 10.15.25.0/24?

I guess the easiest way would be just to see the FWSM configuration. Maybe you can post it (unless its too big or contains too much to parse through) or send it through private message.

For some reason the forums are facing problems with attachement files at the moment so dont use that. I couldnt open them. (I think the problem is beeing looked into though)

- Jouni

[gignet200]

Just sent you a PM.  Thank you.

[Jouni Forss]

Hi,

I read your PM and am still a bit confused as to the routing in this case.

• On the core device which hosts the FWSM are we talking just about Global Routing Table or are there any VRFs involved?
• Looking at the FWSM configuration, what is the interface behind which the remote network 192.168.1.0/24 is located?
  • I see that in your PM you have added and removed some configurations so I just want to confirm
  • For example there was a "route inside 192.168.1.0 255.255.255.0 192.168.0.x" where the gateway IP address is not located on the "inside" interface at all but another interface on the FWSM.

If I have understood the situation correctly it almost seems like the situation is as follows:

• FWSM interface "inside" is facing the remote network 192.168.1.0/24
  • This is also why it would answer to ICMP as its actually facing the remote network
• ICMP Echo messages from the remote network 192.168.1.0/24 go as follows
  • ICMP reaches the local Core and heads straight to host 10.15.25.x
  • ICMP reaches the 10.15.25.x host
  • Host 10.15.25.x sends the traffic to its default gateway = FWSM interface IP? (not the core so asymmetric routing)
  • FWSM blocks the traffic as its not allowed at the moment to pass traffic back through the same "inside" interface where the traffic came from

The solution might be as simple as adding "same-security-traffic permit intra-interface"

But I am not sure if I got the whole setup right. Been handling so many things today that I might have just missed something.

- Jouni

[gignet200]

Hey Jouni.  I do not have any VRFs, and I am transporting these private subnets via OSPF from one side to another. 

Physical Topology

LAN - FWSM/MSFC - CORE ROUTER - MPLS CLOUD - REMOTE CORE ROUTER - CiscoFW - REMOTE LAN

FWSM inside networks:

192.168.0.0/24

10.15.25.0/25

Remote Router Inside Networks:

192.168.1.0

10.15.35.0

The Core routers have an IP in each LAN range, and I am injecting them into OSPF so they get propogated to the remote ends.

If you look at my config you will see i have "same-security-traffic permit intra-interface" applied.  My FWSM NAT rules say for any traffic towards 192.168.1.0/10.15.35.0 do not NAT. 

Currently I can ping from the FWSM LAN (10.15.25.0) into the REMOTE LAN (10.15.35.0), but I cannot ping the other way around.   I hope i make sense.  What do you think thus far?  Thx

[Jouni Forss]

Hi,

In the configuration you sent me you had the configuration line "same-security-traffic permit inter-interface" not "intra-interface"

• inter-interface  = Is used to permit traffic betweem 2 interfaces with equal security-level
• intra-interface = Is used to permit traffic entering and leaving the same interface.

What I wonder about this situation is that if you have all the networks in the global routing table on the local side and you have created a L3 interface which belongs to the 10.15.25.0/24 network then wouldnt this mean that traffic from the remote network destined for that network would reach the hosts before actual getting to the FWSM?

I guess I would really have to see the current (unchanged) configurations of the 6503 and the FWSM and possinbly the local core router to get some sense of this for myself. (If that is even possible)

It might be that you are not doing anything wrong and I am just missing stuff trying to look at some many things at a time.

Think I'll take a short coffee break atleast. Been working 9 hours + all this time on the forums

- Jouni

[gignet200]

I think you need more than a short break after a 9+ hour run   The 192.168.0.0/24 ; 10.15.25.0/24 networks sit behind the FWSM and are using the FWSM as their default gateway.   Same goes for the remote end.  The remote side has networks 192.168.1.0/24 & 10.15.35.0/24 which sit behind the Cisco IOS router/firewall and are using that as their default gateway.  so technically these hosts must go through the FWs at each end to traverse the network.  The fact that I can ping a SVI on the MSFC from the remote end, but not anything passed the FWSM is what is confusing.  I will PM you some more of my configs, so when you get a chance please look at them.  thanks again

[gignet200]

Here is the physical topology of this setup.  Hopefully it clears up some confusion.  I have also PM'd you the full configs for the FWSM and Cisco2851.  Thanks!

[Jouni Forss · Accepted solution]

I would have personally been most interested in the configuration of the actual Cisco 6503 device.

Anyway, here is one thing that seems off to me on the FWSM configuration

You have the following interface and its network/subnet

ip address tosa-lan 192.168.0.245 255.255.255.0

Yet you have the following routes configured to be found behind "inside" interface WHILE at the sametime the gateway IP address of those routes isnt located on the "inside" interface but the "tosa-lan" interface

route inside 10.15.35.0 255.255.255.0 192.168.0.254 1

route inside 192.168.1.0 255.255.255.0 192.168.0.254 1

route inside 192.168.8.0 255.255.255.0 192.168.0.254 1

route inside 192.168.5.0 255.255.255.0 192.168.0.254 1

route inside 192.168.13.0 255.255.255.0 192.168.0.247 1

route inside 192.168.8.0 255.255.255.0 192.168.0.247 1

So something not correct there.

This is why I would want to see teh configurations of the 6503. I have had hard time to see what FWSM interface is actually the one that should lead to the remote site.

- Jouni

[gignet200]

Hey Jouini.  That is correct.  Right now I have 10.15.25.245/192.168.0.245 set on the FWSM.  This is a production network and I have another firewall in place (IOS FW) that has .1 and is doing the routing for customer traffic.  So the host I am working on (10.15.25.89) has its default gw set to .245  I will shoot you the 6503 config now.  Also the static routes you see are in place to allow services to flow to internal subnets.  I learned without these static routes on both ends, you can ping but cannot access any services.

[gignet200]

Jouini, have you had a chance to look over the configs i sent you?  Anything stand out?  Thx

[gignet200]

As a simple test, I have a windows box on each end that I am trying to RDP to.  I have the following ACL applied, and I see the hit counter go up, but still cant connect.  Is there something else thats maybe automatically blocking this?  Thx

When I try to RDP from 10.15.25.89 to 10.15.35.222, I see the counter raise each time, so its definatley hitting it, but no connection. 

access-list ALLOWED extended permit tcp 10.15.25.0 255.255.255.0 10.15.35.0 255.255.255.0 eq 3389 (hitcnt=1)

access-list ALLOWED extended permit tcp 10.15.25.0 255.255.255.0 10.15.35.0 255.255.255.0 eq https (hitcnt=4)

access-list ALLOWED extended permit udp 10.15.25.0 255.255.255.0 10.15.35.0 255.255.255.0 (hitcnt=0)

access-list ALLOWED extended permit icmp 10.15.25.0 255.255.255.0 10.15.35.0 255.255.255.0 (hitcnt=0)

access-list ALLOWED extended permit ip any any (hitcnt=34)

FWSM# show access-group

access-group ALLOWED in interface inside

[Jouni Forss]

Hi,

I still think there is something wrong with how the traffic is forwarded.

The Static routes on the FWSM pointing towards an interface and gateway IP address that dont match doesnt make sense.

You also said you got another router at the main site which further complicates things from my standpoint. Most of my troubleshooting at work is related to simply checking firewalling problems but also some routing setups. Though our solutions usually follow the practices that we have found to be good its usually easy to pinpoint the reason for a problem when you know the environment.

The problem for me here is that there is so many devices and also many thing I cant see (specifically routing) since there is dynamic routing involved.

I would personally start going through this setup by going hop by hop checking the routing table and confirming the traffic flows symmetrically between the endpoints.

For example to my understanding you are testing on some host on the LAN and its GW is set to the FWSM while others to my understanding are using some other gateway and not using the FWSM at all. So there might be a situation where your test hosts traffic (TCP SYN) is flowing through FWSM and then return traffic is flowing back to the host through the router you mentioned before, therefore the traffic that the host then sends back again to the remote host is interpeted by the FWSM as invalid traffic as it has never seen the SYN ACK from the remote site host.

But again this is guessing game for me without knowing the whole layout of the network.

- Jouni

[gignet200]

We ended up having to setup TCP-STATE-BYPASS on the intrernal networks in question.  Once applied, I was able to access services on all private networks.  Thanks for the help Jouni!

[Jouni Forss]

Hi,

I guess the traffic is still flowing in an asymmetric way. I guess you also had to update the FWSM software? I was looking at the same "set connection advanded-options tcp-state-bypass" configuration but I think it was only for 3.2 software or something silimiliar. It basically evades the problem of the routing at the moment.

I think to truly get the most out of the FWSM it would be good to separate the LAN networks to their own VRFs on the C6503. Then bring each network/VRF to the FWSM with their own Vlan link network. And then configure a link from the FWSM towards the other site.

Though I dont know what your plans are for the other Router/FW device on the network that you were talking about.

- Jouni