Solved: Servers can't talk between subnets

WStoffel1 · ‎04-24-2013

This is on a customer 5520 so I can't post the config. Also I only have ASDM access to it as it's at a remote location and that's how it was setup. So...

Server1 is 172.16.9.112 /22 and needs to talk to Server2 which is 172.16.12.3 /22

NAT control is enabed

packet tracer is showing the implicit deny at the end of the following ACL as dropping, line 37 wasn't there initially, that's what i added....but same result in packet tracer.

access-list inside_acl; 84 elements

access-list inside_acl line 1 extended permit ip host 172.16.11.220 any (hitcnt=0) 0xbfda8ac7

access-list inside_acl line 2 extended permit ip host 172.16.11.5 any (hitcnt=0) 0xe65889e3

access-list inside_acl line 3 extended permit tcp host 172.16.0.0 any eq citrix-ica (hitcnt=0) 0x20f0ea4e

access-list inside_acl line 4 extended permit udp host 172.16.0.0 any eq 1494 (hitcnt=0) 0xf226e3e8

access-list inside_acl line 5 extended permit tcp host 172.16.11.28 any eq www (hitcnt=1032) 0x26d12da9

access-list inside_acl line 6 extended permit tcp host 172.16.30.5 any eq www (hitcnt=11259) 0xc4065447

access-list inside_acl line 7 extended permit tcp host 172.16.11.241 any eq www (hitcnt=0) 0x59447315

access-list inside_acl line 8 extended permit tcp host 172.16.11.2 any eq www (hitcnt=1288935) 0x849a032f

access-list inside_acl line 9 extended permit tcp host 172.16.11.1 any eq www (hitcnt=1106) 0x839a574e

access-list inside_acl line 10 extended permit tcp host 172.16.11.221 any eq www (hitcnt=0) 0xcc144af0

access-list inside_acl line 11 extended permit tcp host 172.16.11.28 any (hitcnt=60) 0x7402eec3

access-list inside_acl line 12 extended permit tcp host 172.16.30.9 any (hitcnt=0) 0xdd7b0946

access-list inside_acl line 13 extended permit ip host 172.16.11.28 any (hitcnt=2) 0x0a982bbc

access-list inside_acl line 14 extended permit ip host 172.16.30.9 any (hitcnt=0) 0xfab9a6b0

access-list inside_acl line 15 extended permit tcp host 172.16.11.16 any eq www (hitcnt=0) 0x3bed5f92

access-list inside_acl line 16 extended permit tcp host 172.16.11.16 any eq ftp (hitcnt=0) 0xf0018dd5

access-list inside_acl line 17 extended permit tcp host 172.16.30.230 any eq www (hitcnt=0) 0x954a8b12

access-list inside_acl line 18 extended permit tcp host 172.16.30.78 any eq www (hitcnt=539912) 0xa12b8386

access-list inside_acl line 19 extended permit ip host 172.16.12.30 any (hitcnt=1132) 0x6ed40f4e

access-list inside_acl line 20 extended permit icmp any any (hitcnt=9287635) 0x35273a55

access-list inside_acl line 21 extended permit ip host 172.16.11.0 64.90.226.0 255.255.255.0 (hitcnt=0) 0x675672d1

access-list inside_acl line 22 extended permit tcp host 172.16.11.0 64.90.226.0 255.255.255.0 (hitcnt=0) 0x3f499c6d

access-list inside_acl line 23 extended permit udp host 172.16.11.0 64.90.226.0 255.255.255.0 (hitcnt=0) 0xeac29479

access-list inside_acl line 24 extended permit tcp host 172.16.11.0 64.90.227.0 255.255.255.0 (hitcnt=0) 0x02d65bde

access-list inside_acl line 25 extended permit udp host 172.16.11.0 64.90.227.0 255.255.255.0 (hitcnt=0) 0xd7ccf6af

access-list inside_acl line 26 extended permit tcp host 172.16.11.0 24.90.235.0 255.255.255.0 (hitcnt=0) 0x7819c33d

access-list inside_acl line 27 extended permit udp host 172.16.11.0 24.90.235.0 255.255.255.0 (hitcnt=0) 0x13d841e6

access-list inside_acl line 28 extended permit tcp host 172.16.11.43 any eq www (hitcnt=424534) 0x7f47ddec

access-list inside_acl line 29 extended permit udp host 172.16.11.43 any eq www (hitcnt=0) 0x195e9f62

access-list inside_acl line 30 extended permit ip host 172.16.11.43 any (hitcnt=852263) 0x0cc9ced9

access-list inside_acl line 31 extended permit tcp host 172.16.11.229 any eq www (hitcnt=25584343) 0x92a78b92

access-list inside_acl line 32 extended permit ip host 172.20.11.11 any (hitcnt=0) 0x2d818222

access-list inside_acl line 33 extended permit ip any any (hitcnt=1816078595) 0x2ee35316

access-list inside_acl line 34 extended permit tcp any any (hitcnt=0) 0xab114b84

access-list inside_acl line 35 extended permit ip host 172.16.11.239 any (hitcnt=0) 0x283d9008

access-list inside_acl line 36 extended permit ip object-group VPN 172.20.8.0 255.255.255.0 log debugging interval 300 0x377c0d70

access-list inside_acl line 36 extended permit ip 172.16.20.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xb0f69b3c

access-list inside_acl line 36 extended permit ip 172.16.25.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xee89c3ce

access-list inside_acl line 36 extended permit ip 172.16.30.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xc9279d34

access-list inside_acl line 36 extended permit ip 172.16.35.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x8ff3ba27

access-list inside_acl line 36 extended permit ip 172.16.40.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x9054b2cc

access-list inside_acl line 36 extended permit ip 172.16.50.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x566922f0

access-list inside_acl line 36 extended permit ip 172.16.60.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x730a2aa4

access-list inside_acl line 36 extended permit ip 172.16.65.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xe4a48265

access-list inside_acl line 36 extended permit ip 172.16.70.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xfcf2fa02

access-list inside_acl line 36 extended permit ip 172.16.75.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x0be569b4

access-list inside_acl line 36 extended permit ip 172.16.85.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xb288c563

access-list inside_acl line 36 extended permit ip 172.16.95.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x8c26932a

access-list inside_acl line 36 extended permit ip 172.20.11.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x403ec994

access-list inside_acl line 36 extended permit ip 172.20.25.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xc093a553

access-list inside_acl line 36 extended permit ip 172.20.30.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xd12b35c8

access-list inside_acl line 36 extended permit ip 172.20.40.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x556c15a2

access-list inside_acl line 36 extended permit ip 172.20.50.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x8282d11e

access-list inside_acl line 36 extended permit ip 172.20.60.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xbe179bea

access-list inside_acl line 36 extended permit ip 172.20.85.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xf921a60c

access-list inside_acl line 36 extended permit ip 172.20.95.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x384d05e2

access-list inside_acl line 36 extended permit ip 192.168.30.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x547413b9

access-list inside_acl line 36 extended permit ip 192.168.50.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x423dfafb

access-list inside_acl line 36 extended permit ip 10.10.10.0 255.255.255.224 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xc411a500

access-list inside_acl line 36 extended permit ip 172.16.61.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x9ba95ded

access-list inside_acl line 36 extended permit ip 192.168.20.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x006d800b

access-list inside_acl line 36 extended permit ip 172.20.90.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x2cf56d17

access-list inside_acl line 36 extended permit ip 172.16.90.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xe72150f1

access-list inside_acl line 36 extended permit ip 172.16.86.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x0fa58760

access-list inside_acl line 36 extended permit ip 172.16.87.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xe8cbf2b2

access-list inside_acl line 36 extended permit ip 172.16.100.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x573669bc

access-list inside_acl line 36 extended permit ip 172.21.11.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x314a8ebd

access-list inside_acl line 36 extended permit ip 172.16.37.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xe715a700

access-list inside_acl line 36 extended permit ip 172.16.4.0 255.255.252.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x0095535a

access-list inside_acl line 36 extended permit ip 172.16.36.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x9c8aca24

access-list inside_acl line 36 extended permit ip 172.16.12.0 255.255.252.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xab3a59d4

access-list inside_acl line 36 extended permit ip 128.17.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x7c84da24

access-list inside_acl line 36 extended permit ip 128.18.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x59c77c1d

access-list inside_acl line 36 extended permit ip 128.19.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x89c0a16d

access-list inside_acl line 36 extended permit ip 128.20.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x50ee57cb

access-list inside_acl line 36 extended permit ip 128.21.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xd256f545

access-list inside_acl line 36 extended permit ip 128.22.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x4c348b4c

access-list inside_acl line 36 extended permit ip 128.23.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x5fb7f057

access-list inside_acl line 36 extended permit ip 128.24.0.0 255.255.0.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x6d808e91

access-list inside_acl line 36 extended permit ip 172.16.8.0 255.255.252.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xe838eb75

access-list inside_acl line 36 extended permit ip 172.16.91.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0xa0c922fd

access-list inside_acl line 36 extended permit ip 172.16.80.0 255.255.252.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x2f50bb63

access-list inside_acl line 36 extended permit ip 172.16.88.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x85a549b4

access-list inside_acl line 36 extended permit ip 172.20.8.0 255.255.255.0 172.20.8.0 255.255.255.0 log debugging interval 300 (hitcnt=0) 0x0d481fdf

access-list inside_acl line 37 extended permit ip host A-172.16.12.3 host A-172.16.9.112 (hitcnt=0) 0x3944cd4a

I also added a nat exempt rule.

If anyone has any ideas by all means but I do have a question, because although packet tracer shows it dropping becuase of an ACL, is it still possible for it to be a routing issue (a missing static route)?

Thank you.

Jouni Forss · ‎04-24-2013

Hi,

If the server 172.16.9.112 is using some other device other than the ASA as its default gateway it would seem to me that the ASA would be totally out of the picture.

As this would mean that both of the server LAN networks would be using some gateway behind the ASA "inside" interface. So it could be possible that the ASA has nothing to do with the traffic between these servers.

Seems the server 172.16.9.112 is using gateway 172.16.11.254

At the sametime the ASA configurations states that the network 172.16.12.0/22 is found behind the gateway 172.16.11.251.

- Jouni

View solution in original post

Jouni Forss · ‎04-24-2013

Hi,

Naturally I would hope for more information. I do understand that you cant share a full configuration but it just makes this harder to solve. It's like being asked so solve a network problem but not allowed access to any of the devices

Could you possibly share the output of the "packet-tracer"?

And output of "show route"

You can actually run the CLI format commands on the ASDM from Tools -> Command Line Interface

Though to me it seems the above ACL is already copied from CLI format configuration and not ASDM.

Can you check if the souce and destination interfaces have equal "security-level"? If they do add "same-security-traffic permit inter-interface"

One thing that also confuses me is that according to the IP address and network masks that you give you have the following 2 networks

172.16.8.0/22

172.16.12.0/22

The ACL statement above says the source as 172.16.12.3 which belongs to network 172.16.12.0/22

At the same time the same ACL has rules which define source address for example as 172.16.11.2 which belongs to the network 172.16.8.0/22

So either the new rule is on the wrong ACL or the 2 networks are behind the same interface of the ASA? Or dud I missunderstand something.

- Jouni

WStoffel1 · ‎04-24-2013

Jeez you're fast! And yes i know i'm handicapping not putting the config in...the ACL output was from the command line interface in ASDM. Problem is crypto rsa needs to be zeroized and regenerated but i can't do that from asdm.

Ok, backwards order, they are both behind the inside interface. It's two physical locations. I'm still trying to figure out the routing

But yes, the 172.16.8.0/22 network contains the 172.16.9.112 server and the

172.16.12.0/22 network contains the 172.16.12.3 server and thinking about it now, I dont have the gateway info with me, just the IPs.

Same security traffic inter AND intra interface is not enabled. Not something I was going to do just on the fly, especially since it appears other INTRA interface traffic is working...

Result of the command: "packet-tracer input inside tcp 172.16.12.3 1065 172.16.9.112 80 det"

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 172.16.8.0 255.255.252.0 inside

Phase: 3

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xc87ec9d0, priority=111, domain=permit, deny=true

hits=68821129, user_data=0x0, cs_id=0x0, flags=0x4000, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

Thinking more about it....i edited tunnel group and public IP's out, see attached

Thank you sir!

Jouni Forss · ‎04-24-2013

Hi,

The "packet-tracer" command will most probably fail because the traffic is entering the same interface that its supposed to leave.

same-security-traffic permit intra-interface

Will probably change the output of the "packet-tracer" command.

But it seems to me that its not your only problem in this situations.

What I find potentially problematic in this situation is that the "inside" is directly connected to the network 172.16.8.0/22

The other network 172.16.12.0/22 is routed towards some other routers behind the "inside" interface.

We will run into problems if hosts on the 172.16.8.0/22 are using the ASA "inside" interface as a default gateway.

What will basically happen is asymmetric routing

Host on network 172.16.8.0/22 will start to open a TCP connection with TCP SYN towards the host in network 172.16.12.0/22
The TCP SYN arrives on the ASA if ASA is set as the default gateway for the network 172.16.8.0/22
Provided the ASA is added with NAT and ACL configuration to allow this traffic it will forward it back to the router behind "inside" interface
Host on the network 172.16.12.0/22 receives the TCP SYN and sends TCP SYN,ACK back to the host that started the TCP connection negotiation
When that TCP SYN,ACK reaches the router behind ASA "inside" interface the router will naturally see the network 172.16.8.0/24 as directly connected network. It therefore has no need to route the traffic to the ASA where it came from
ASA will still be waiting for the TCP SYN,ACK
The host on 172.16.8.0/22 has already received the TCP SYN,ACK directly from the router as we mentioned above
The host on 172.16.8.0/22 will send the final TCP ACK to finalize the TCP connection negotiation and sends it to its default gateway ASA.
ASA is still waiting to see the TCP SYN,ACK and sees the TCP ACK. It blocks this traffic and the TCP connection forming will fail.

If the above is indeed the network layout your only option to my knowledge is to configure TCP State Bypass for some of these connections which effectively is something that removes an essential part of ASA operation looking at the State of the TCP connections. In other words it wont care that it hasnt seen the TCP SYN,ACK and lets the TCP ACK go through and lets the servers communicate with eachother.

I cant say that this is the situation 100%. You will have to confirm what the network 172.16.8.0/22 is using as their gateway first.

If you want to read up on teh TCP State Bypass, check some of these links

http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a0080b2d922.shtml

http://nat0.net/asa-state-bypass/

http://ngthomas.co.uk/Blog/tcp-state-bypass-on-a-cisco-asa/

- Jouni

WStoffel1 · ‎04-24-2013

I knew it couldn't be easy! Thank you. I'll need some time to digest this.

I will tell you that the devices on the 172.16.8.0 network use different gatways. The server 172.16.9.112 uses 172.16.11.254 and another device at 172.16.11.228 uses 172.16.11.251, neither of which is the firewall

I don't however know if anything is actually using the inside interface as their gateway. It's possible so I'll have to do some legwork.

Jouni Forss · ‎04-24-2013

Hi,

If the server 172.16.9.112 is using some other device other than the ASA as its default gateway it would seem to me that the ASA would be totally out of the picture.

As this would mean that both of the server LAN networks would be using some gateway behind the ASA "inside" interface. So it could be possible that the ASA has nothing to do with the traffic between these servers.

Seems the server 172.16.9.112 is using gateway 172.16.11.254

At the sametime the ASA configurations states that the network 172.16.12.0/22 is found behind the gateway 172.16.11.251.

- Jouni

WStoffel1 · ‎04-24-2013

Changed the gateway on 172.16.9.112 to 251 and guess what? Not only did the traffic start working correctly, but the local guy now mentions to me that the server can now get to the internet. In other words it was the wrong gateway to begin with and the server wasn't even online, except on the local segment. Another case of not having all the pieces to the puzzle.

Aye yi yi yi yi. Thank you.

Jouni Forss · ‎04-24-2013

Glad to hear its working now

- Jouni