Very odd issue...need help

burleyman · ‎10-20-2015

I have an ASA that is running 9.1(1)

I have customers that use an application that goes to my outside IP address using TCP port 15678 and redirects it to a server in the DMZ with an IP address of 10.2.1.22.

Here is the configuration for that to happen.

interface GigabitEthernet0/1
nameif outside
security-level 0
ip address 128.1.1.2 255.255.255.248

interface GigabitEthernet0/2
description connection to DMZ interface
nameif DMZ
security-level 50
ip address 10.2.1.254 255.255.255.0

object network OBJ-10.2.1.22_32
host 10.2.1.22

object service OBJ-TCP-15678
service tcp source eq 15678

access-list outside_access_in extended permit tcp any4 object OBJ-10.2.1.22_32 eq 15678

nat (DMZ,outside) source static OBJ-10.2.1.22_32 interface service OBJ-TCP-15678 OBJ-TCP-15678

route outside 0.0.0.0 0.0.0.0 128.1.1.1

Now the very odd issue.

all customers that have this setup are running with no issues except one.

They are coming from an IP address of 50.50.50.54 and they do not get connected to the server on the DMZ and here is what I see in the logs.

6|Oct 20 2015|10:21:32|106015|10.2.1.22|15678|50.50.50.54|61991|Deny TCP (no connection) from 10.2.1.22/15678 to 50.50.50.54/61991 flags SYN ACK on interface inside

Why does it say "SYN ACK on interface inside"? and not DMZ? I think this is where the issue lies but I don't know what is causing it. What should I be looking at?

Mike

Jon Marshall · ‎10-20-2015

Mike

What does the routing table look like ?

What does a packet tracer show ie. -

"packet-tracer input outside tcp 50.50.50.54 12345 <public IP of server> 15678"

Jon

burleyman · ‎10-20-2015

ASA# show route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default, U - per-user static route, o - ODR
       P - periodic downloaded static route

Gateway of last resort is 128.1.1.1 to network 0.0.0.0

C    172.16.1.0 255.255.255.0 is directly connected, inside
C    60.60.60.0 255.255.255.252 is directly connected, backup
C    128.1.1.0 255.255.255.248 is directly connected, outside
C    10.2.1.0 255.255.255.0 is directly connected, DMZ
S    10.2.2.0 255.255.255.0 [1/0] via 128.1.1.1, outside
                            [1/0] via 60.60.60.1, outside
S    10.169.3.0 255.255.255.0 [1/0] via 128.1.1.1, outside
                              [1/0] via 60.60.60.1, outside
S    192.168.2.0 255.255.255.0 [1/0] via 128.1.1.1, outside
                               [1/0] via 60.60.60.1, outside
S*   0.0.0.0 0.0.0.0 [1/0] via 128.1.1.1, outside

ASA# packet-tracer input outside tcp 50.50.50.54 12345 10.2.1.22 15678

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 10.2.1.0 255.255.255.0 DMZ

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit tcp any4 object OBJ-10.2.1.22_32 eq 15678
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
nat (DMZ,outside) source static OBJ-10.2.1.22_32 interface service OBJ-TCP-15678 OBJ-TCP-15678
Additional Information:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: DMZ
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Not sure why the drop.....

Mike

Jon Marshall · ‎10-20-2015

The drop is because you used the private IP of the server and not the public IP.

However I should have asked for it the other way round.

Can you run this -

"packet-tracer input DMZ tcp 10.1.2.22 15678 50.50.50.54 12345"

Jon

burleyman · ‎10-20-2015

ASA# packet-tracer input DMZ tcp 10.1.2.22 15678 50.50.50.54 12345

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group DMZ_access_in in interface DMZ
access-list DMZ_access_in extended permit ip any any
Additional Information:

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 5144622, packet dispatched to next module

Result:
input-interface: DMZ
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

When I do the command show conn here is what I get for a site that works and one for the site that does not.

Works

TCP outside 75.75.75.225:1054 DMZ 10.2.1.22:15678, idle 0:00:00, bytes 766, flags UIOB

Does not work

TCP outside 50.50.50.54:61525 DMZ 10.2.1.22:15678, idle 0:00:01, bytes 0, flags SaAB

Mike

Jon Marshall · ‎10-20-2015

Mike

The packet tracer output looks okay.

The "sh conn" output ties up with your logs ie. the flags for the connection that doesn't work indicates a SYN packet has been received by the server but obviously if the SYN ACK is going to the inside interface then the remote device doesn't receive it and so never sends the ACK.

So something in your configuration must be sending that traffic to the inside but it isn't your route table as far as I can see.

Sorry to be a pain but can you run the original packet tracer again but this time use the public IP of the server.

Jon

burleyman · ‎10-21-2015

No bother and thank you for your help. I went over the config again and nothing that should send the traffic to the inside interface. In fact this is pretty much the only rule in the ASA. It is only there to do pretty much this one thing.

ASA# packet-tracer input DMZ tcp 128.1.1.2 15678 50.50.50.54 12345

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group DMZ_access_in in interface DMZ
access-list DMZ_access_in extended permit ip any any
Additional Information:

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 5180851, packet dispatched to next module

Result:
input-interface: DMZ
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Jon Marshall · ‎10-21-2015

Mike

To say it is odd is an understatement :-)

I can't see why it would send that traffic to the inside interface.

I am happy to provide a second pair of eyes on the configuration if you want to post it but you know what you are doing so I doubt I will spot anything.

Jon

burleyman · ‎10-21-2015

I know.....These are fun and frustrating all at the same time.

I could post the config later but I don't think it would help very much because it is pretty much and default route and the config I posted, very basic. And it is working for everyone else but just this one customer and possibly a couple others but 50 to 100 others work just fine.

Thanks,

Mike

Rishabh Seth · ‎10-22-2015

Hi Mike,

From the syslogs it looks like the SYN ACK is coming on the inside interface.

It is possible that you have some routing issue in your internal network due to which the reply traffic is coming to the inside interface.

Can you try to do traceroute from internal machine with IP 10.2.1.22 for any public IP and see how traffic flows.

Do share your findings.

Thanks,

R.Seth

burleyman · ‎10-23-2015

A reboot of the server resolved the issue. I believe the DMZ server was for some reason send that traffic inside. I think it was the application and the transactions it does got messed up and routed the traffic for only a few transactions to the inside.

Thank you everyone for your help.

Mike

Rishabh Seth · ‎10-23-2015

Glad that the issue is resolved. :)

Thanks,

R.Seth