topic Re: Firewall blocking traffic between VLANS/subnets in Network Security

Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 11 Mar 2019 21:22:39 GMT

Here's the problem we've encountered, was hoping you guys could help me debug what the heck is going on:

In the below diagram, computers on 10.20.1.x cannot ping anything on the 10.20.3.x subnet (Including Firewall B).
However, Firewall A can ping Firewall B and the .3.x server so it's not a VLAN routing problem.

The strange thing is that according to the ASDM Packet trace on Firewall A (Screenshot), packets are allowed to travel freely between the internal pc's and the .3.x subnet servers.

I'm positive the fault lies with Firewall A, but the running config file is way too big to post here so I'll try dig out what looks relevant.

We have the following NAT rule which I would have though was the solution but that doesn't seem to help:

static (inside,bt_dmz) 10.20.1.0 10.20.1.0 netmask 255.255.255.0

(bt_dmz is the interface on Firewall A that runs through to the 10.20.3.x subnet)

We also have icmp permitted:

icmp permit 10.20.1.0 255.255.255.0 inside

icmp permit 10.20.2.0 255.255.255.0 dmz

icmp permit 10.20.3.0 255.255.255.0 bt_dmz

Access list rule:

access-list outside_access_in_dmz extended permit icmp any any

If Firewall A can ping Firewall B and the server, it feels like routing between VLANS is ok but why then can I not ping the server on that VLAN from my internal PC? Could there be a conflicting rule?

Firewall blocking traffic between VLANS/subnets

varrao — Fri, 09 Sep 2011 11:50:49 GMT

Hi Wez,

I was answering to your thread but I guess you moved it by then

Well I woudl say take captures and analyse, thats the best thing. Check where the packets are getting dropped. Also you woudl need a nat statement for translating the source traffic On firewall 1, something like this:

nat (inside) 10 10.20.1.0 255.255.255.0

global (dmz) 10 interface

On the firewall 2, you shoudl have the following:

static (inside,dmz) 10.20.1.0 10.20.1.0

I would request you to kindly try it and take captures. On firewall 2 as well you need the access-list going from dmz to inside, so plz make sure.

For captures:

https://supportforums.cisco.com/docs/DOC-17814

Hope this helps.

Thanks,

Varun

Firewall blocking traffic between VLANS/subnets

KingPrawns — Fri, 09 Sep 2011 12:58:09 GMT

Hey, sorry about that. I posted in routing but fixed an issue soon after which made it seem more like a firewall issue than a routing one so I recreated the post here.

With Firewall B, I'm not sure it needs the change you recommended as it's only role is to serve traffic between .3.x servers and the internet. It doesn't actually fall between any internal traffic. Though I might be wrong on that one.

With Firewall A, we have the current rules:

static (inside,bt_dmz) 10.20.1.0 10.20.1.0 netmask 255.255.255.0

access-list inside_nat0_outbound extended permit ip any 10.20.1.16 255.255.255.240

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 101 0.0.0.0 0.0.0.0

global (outside) 101 interface

global (BT_Outside) 103 interface (Redundant)

global (bt_dmz) 102 interface

So if I rejig your rules, should I still add the following? :

nat (inside) 101 10.20.1.0 255.255.255.0

And does this make the global rule ( global (bt_dmz) 10 interface ) redundant?

Firewall blocking traffic between VLANS/subnets

varrao — Fri, 09 Sep 2011 13:04:17 GMT

Hi Wez,

Yeah I had a look at it again, and i guess if the traffic for the server is not going through the 2nd firewall, then you don't need any config on it. From the config that you have , i can only see one thing that needs to be added:

global (bt_dmz) 101 interface

You can try it, if it still does not work, take captures and trace the packets.

Thanks,

Varun

Re: Firewall blocking traffic between VLANS/subnets

KingPrawns — Fri, 09 Sep 2011 14:08:10 GMT

Here are the results from the packet capture wizard. It looks like packets are going one way but not coming back the other.

I'm guessing that means there are access rules allowing traffic from my pc to the server, but not in the other direction?

Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 12 Sep 2011 09:15:18 GMT

What does global (bt_dmz) 101 interface actually do? As far as I understood, it assigns one of the physical ports on the box as an interface.

The outside interface assigned to 101 is already used to connect out to the internet. 102 is where the bt_dmz connects up to.

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 10:28:08 GMT

Hi Wez,

The statement is for patting the inside users to the ip address of the bt_dmz interface:

nat (inside) 101 0.0.0.0 0.0.0.0

global (bt_dmz) 101 interface

It is the nat ID's that needs to be matched, 101 is the nat id for the global statement corresponding to the one in the nat statement.

As you can see in the captures, when you ping, the echo request is going fine through the firewall, but we do not see and replies coming back for it, I would suggest you use the global statetment suggested and then take the captures again.

Let me know i you have any issues.

Thanks,

Varun

Re: Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 12 Sep 2011 11:13:58 GMT

Added the rule and re-pinged, still have the same problem. The request is going one way, but not the other.

New globals/nats look like this:

     global (outside) 101 interface
     global (BT_Outside) 103 interface
     global (bt_dmz) 101 interface
     nat (inside) 0 access-list inside_nat0_outbound
     nat (inside) 101 0.0.0.0 0.0.0.0

Ran the packet tracer tool on the asa, From inside to bt_dmz, it works as expected but the other way I get an access list failure. (Though I'm not sure it is because I have the wrong interface selected) - Screen shot

The access list it drops on is the default any - any - deny rule in the inside interface.

Could it be something like a static route that had been set up ?

We had once tried to get the BT_dmz to go out via BT_outside interface, but abandoned that plan. I don't think there are any rules left that would cause issue though.

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 11:38:07 GMT

Can you provide me the access-list that you have applied on the inside interface??

Thanks,

Varun

Re: Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 12 Sep 2011 13:08:39 GMT

Sure,

access-list outside_access_in_dmz extended permit icmp any any

access-list outside_access_in_dmz extended permit tcp any object-group EmailServers eq smtp

access-list outside_access_in_dmz extended permit object-group WebsiteTraffic any object-group WebServers

access-list outside_access_in_dmz extended permit udp any host 10.20.1.5 eq sip

access-list Internal standard permit 192.168.2.0 255.255.255.0

access-list Internal standard permit 10.20.1.0 255.255.255.0

access-list Internal standard permit 10.20.2.0 255.255.255.0

access-list Internal standard permit 10.1.1.0 255.255.255.0

access-list Internal standard permit 10.1.10.0 255.255.255.0

access-list inside_access_in extended permit ip any any

access-list dmz_to_inside extended permit ip any any

access-list inside_nat0_outbound extended permit ip any 10.20.1.16 255.255.255.240

access-list vlan1_bt_dmz_access_in extended permit ip any any

access-list inside_access_in_1 extended permit ip any 10.20.3.0 255.255.255.0

access-list inside_access_in_1 extended permit ip any any

access-list BT_Outside_access_in extended permit icmp any any

Nothing is standing out as wrong from what I can see

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 13:12:01 GMT

Hi Wez,

Could you also send:

show run access-group as well.

thanks,

Varun

Re: Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 12 Sep 2011 13:20:15 GMT

Sorry, missed those.

access-group outside_access_in_dmz in interface outside

access-group inside_access_in_1 in interface inside

access-group BT_Outside_access_in in interface BT_Outside

access-group dmz_to_inside in interface dmz

access-group vlan1_bt_dmz_access_in in interface bt_dmz

I have a big old list of service and network objects if you want them too, though there's nothing really in there relating to either the .3.x subnet or the bt_dmz

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 13:28:24 GMT

Could you please add this:

access-list vlan1_bt_dmz_access_in extended permit icmp any any

this should resolve the icmp issue and check packet-tracer again.

Thanks,

Varun

Re: Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 12 Sep 2011 13:49:05 GMT

Added that in, and re-ran the packet capture (Results) but with little success.

Would it help if I private messaged you the full running config?

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 13:51:58 GMT

Sure Wez, go ahead. You can PM me the config.

-Varun

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 14:36:58 GMT

Hi Wez,

doubt a few things, can you plz delete the statement:

static (inside,dmz) 10.20.1.0 10.20.1.0

and try again, if it does not work, i would need the output of:

packet-tracer input inside icmp 10.20.1.161 0 8 10.20.3.151 detailed

taken from the cli and not the wizard.

Thanks,

Varun

Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 12 Sep 2011 14:44:08 GMT

I'm a little worried about deleting static (inside,dmz) 10.20.1.0 10.20.1.0 since I'm sure that relates to our existing servers (10.20.2.x) which we use on a regular basis. Would removing it stop us from connectiong between inside and dmz ?

Here is the output before removing the rule:

# packet-tracer input inside icmp 10.20.1.161 0 8 10.20.13.151 detailed
Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd555a118, priority=1, domain=permit, deny=false
        hits=34883336, user_data=0x0, cs_id=0x0, l3_type=0x8
        src mac=0000.0000.0000, mask=0000.0000.0000
        dst mac=0000.0000.0000, mask=0000.0000.0000
Phase: 2
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside
Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in_1 in interface inside
access-list inside_access_in_1 extended permit ip any any
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd5662268, priority=12, domain=permit, deny=false
        hits=1025011, user_data=0xd5662228, cs_id=0x0, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0
Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd555c2e0, priority=0, domain=permit-ip-option, deny=true
        hits=1028573, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0
Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd555b5e8, priority=66, domain=inspect-icmp-error, deny=false
        hits=3541, user_data=0xd553c368, cs_id=0x0, use_real_addr, flags=0x0, pr                                                                              otocol=1
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0
Phase: 7
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
static (inside,dmz) 10.20.1.0 10.20.1.0 netmask 255.255.255.0
  match ip inside 10.20.1.0 255.255.255.0 dmz any
    static translation to 10.20.1.0
    translate_hits = 561088, untranslate_hits = 18184
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd5619e40, priority=5, domain=host, deny=false
        hits=1027904, user_data=0xd56198c0, cs_id=0x0, reverse, flags=0x0, proto                                                                              col=0
        src ip=10.20.1.0, mask=255.255.255.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0
Phase: 8
Type: NAT
Subtype:
Result: DROP
Config:
nat (inside) 101 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 101 (82.20.64.129 [Interface PAT])
    translate_hits = 463950, untranslate_hits = 118304
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd5616e00, priority=1, domain=nat, deny=false
        hits=788736, user_data=0xd5616d60, cs_id=0x0, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0
Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 15:11:22 GMT

Hi Wez,

I see that the traffic is hitting the static statement instead of the nat statement. and if you remember my first post:

Varun Rao wrote:
I was answering to your thread but I guess you moved it by then 
Well  I woudl say take captures and analyse, thats the best thing. Check  where the packets are getting dropped. Also you woudl need a nat  statement for translating the source traffic On firewall 1, something  like this:
nat (inside) 10 10.20.1.0 255.255.255.0
global (dmz) 10 interface
On the firewall 2, you shoudl have the following:
static (inside,dmz) 10.20.1.0 10.20.1.0
I  would request you to kindly try it and take captures. On firewall 2 as 
well you need the access-list going from dmz to inside, so plz make  sure.

So I had suggested you this, so I think thats when you added it. Correct me if I am wrong.

Thanks,

Varun

Re: Firewall blocking traffic between VLANS/subnets

KingPrawns — Mon, 12 Sep 2011 15:55:31 GMT

Sorry, I should have mentioned that I tweaked that to:

static (inside,bt_dmz) 10.20.1.0 10.20.1.0 netmask 255.255.255.0

However, removing that seems to have done the trick! I can rdp and ping that server absolutely fine now.

Second tickbox is getting servers on 10.20.2.x to be able to ping 10.20.3.x.

The packet trace is failing on

Phase: 4

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xd55366e8, priority=110, domain=permit, deny=true

hits=14, user_data=0x0, cs_id=0x0, flags=0x3000, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0

Does that mean I need to add something like access-list dmz_to_bt_dmz extended permit ip any any to the access list?

Also, would I need static (dmz,bt_dmz) 10.20.2.0 10.20.2.0 netmask 255.255.255.0 ?

Firewall blocking traffic between VLANS/subnets

varrao — Mon, 12 Sep 2011 16:08:40 GMT

Excellent!!!!!!!!! So one issue has been knocked out

Coming to the next, you need to just add this:

nat (dmz) 102 0.0.0.0 0.0.0.0

global (bt_dmz) 102 interface

and

access-list dmz_to_inside extended permit icmp any any
same-security-traffic permit inter-interface

and it should work.

If it does, I would need the complete packet-tracer output.

Thanks,

Varun