Access List says allowed, but traffic doesn't pass..Help!

Chris Bloy · ‎11-25-2016

Hi All,

We have just installed a new firewall 5506X running:-

Cisco Adaptive Security Appliance Software Version 9.6(1)

Device Manager Version 7.6(1)

And it has various routes to an MPLS:-

S 140.85.0.0 255.255.0.0 [1/0] via 10.164.115.97, LSP_UPLINK

S 141.143.0.0 255.255.0.0 [1/0] via 10.164.115.97, LSP_UPLINK

S 159.166.0.0 255.255.0.0 [1/0] via 10.164.115.97, LSP_UPLINK

S 162.130.0.0 255.255.0.0 [1/0] via 10.164.115.97, LSP_UPLINK

S 162.130.196.128 255.255.255.255 [1/0] via 62.232.113.153, outside

S 162.130.196.190 255.255.255.255 [1/0] via 62.232.113.153, outside

S 162.130.196.219 255.255.255.255 [1/0] via 62.232.113.153, outside

S 162.130.197.157 255.255.255.255 [1/0] via 62.232.113.153, outside

C 172.16.0.0 255.255.255.0 is directly connected, Associate_VoIP

L 172.16.0.1 255.255.255.255 is directly connected, Associate_VoIP

C 192.168.21.0 255.255.255.0 is directly connected, KEY-CARD-NET

L 192.168.21.1 255.255.255.255 is directly connected, KEY-CARD-NET

S 192.168.72.0 255.255.252.0 [1/0] via 10.164.95.97, LSP_UPLINK

S 192.168.114.0 255.255.255.0 [1/0] via 10.164.95.97, LSP_UPLINK

I can route traffic to the all of the IP's in the routing table bar the 192.168.7x.x subnets even though the access list allows the traffic to pass:-

Phase: 1

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

MAC Access list

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 3

Type: ROUTE-LOOKUP

Subtype: Resolve Egress Interface

Result: ALLOW

Config:

Additional Information:

found next-hop 10.164.95.97 using egress ifc LSP_UPLINK

Phase: 4

Type: ROUTE-LOOKUP

Subtype: Resolve Egress Interface

Result: ALLOW

Config:

Additional Information:

found next-hop 10.164.114.3 using egress ifc ASSOCIATES_VLAN300

Phase: 5

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 8

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 9

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 23292388, packet dispatched to next module

Result:

input-interface: ASSOCIATES_VLAN300

input-status: up

input-line-status: up

output-interface: LSP_UPLINK

output-status: up

output-line-status: up

Action: allow

If i try to ping, i get nothing a all from the server, however, on another firewall running 9.5(2) we don't have the issue, its the same configuration and access-list apart from the internal addressing and that works fine.. I am at a loss as to why it doesn't work on the latest software?

Working on 9.5.2:

TCP Ping [n]: y

Interface: ASSOCIATES_VLAN300

Target IP address: 192.168.72.96

Destination port: [80]

Specify source? [n]:

Repeat count: [5]

Timeout in seconds: [2]

Type escape sequence to abort.

No source specified. Pinging from identity interface.

Sending 5 TCP SYN requests to 192.168.72.96 port 80

from 10.164.95.98, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 91/95/107 ms

Failed on 9.6:-

TCP Ping [n]: y

Interface: ASSOCIATES_VLAN300

Target IP address: 192.168.72.96

Destination port: [80]

Specify source? [n]:

Repeat count: [5]

Timeout in seconds: [2]

Type escape sequence to abort.

No source specified. Pinging from identity interface.

Sending 5 TCP SYN requests to 192.168.72.96 port 80

from 10.164.115.98, timeout is 2 seconds:

?????

Success rate is 0 percent (0/5)

Any Help you can give to diagnose this issue is greatly received.

cofee · ‎11-25-2016

ICMP traffic is not inspected by the firewall so it's possible that you have ICMP inspection turned on or it's explicitly allowed by an acl in the firewall where you are able to ping and not in the one that where it's failing. Since you didn't share interface configuration I can't tell if you are going from a higher to lower or vice versa.

Firewall where ping is failing:

show run policy-map (it will show what protocols are being inspected, tcp and udp are inspected by default)

you can either turn icmp inspection on under the global policy or create a custom class map to only allow specific host.

Chris Bloy · ‎11-25-2016

Hi Cofee, thank you for the reply, but i already have ICMP inspection enabled, but here is the configuration for the interface:-

interface GigabitEthernet1/3.300

vlan 300

nameif ASSOCIATES_VLAN300

security-level 100

ip address 10.164.114.1 255.255.255.128

interface GigabitEthernet1/6

nameif LSP_UPLINK

security-level 60

ip address 10.164.115.98 255.255.255.248

I really have spent day trying to troubleshoot this and have even thought it was a MPLS fault?

Thanks, Chris

cofee · ‎11-25-2016

Since you have already spent this much troubleshooting I am sure you must have tried debugging icmp packets on the local firewall and if you have access to the remote server or the remote firewall did you check if packets are making to the remote firewall/server? if so what do the logs say.

Chris Bloy · ‎11-25-2016

I have run a capture with icmp and https requests based on source of any and destination 192.167.72.96 is the server that the other properties can access. I don't see any traffic exiting the interface to that subnet, but I do on all other subnet.

We don't have access to the destination server or wan as it's run by Verizon, so can only see traffic locally. If i run the show route 192.168.72.96 it correct identifies the static route.

Really appreciate your time and help, but I am beginning to believe it's a software version issue?

we would have to take the site down to prove it with the same release, but it's only this one route that's causing me a headache and it's the finance department!

cofee · ‎11-25-2016

Yeah I think if it's possible to downgrade the new 5560x with 9.5(2) and have the same running configuration will prove your point. If you a chance let me know the outcome, I would like to know.