ASA 5516-x (nat-xlate-failed) NAT failed (Can't connect to and from logical VLANs)

[rodito]

We cannot connect from our different logical VLANs but can connect to inside and DMZ zone. Users on my different logical VLANs the (PSN) network cannot reach any of the other PSN network. inside and DMZ is ok they can reach them.

This is already in the config:

same-security-traffic permit inter-interface
same-security-traffic permit intra-interface

class-map | policy map

class-map sfr-class
match any
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
tcp-inspection
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect ip-options
inspect icmp
class sfr-class
sfr fail-open
class class-default
user-statistics accounting
!
service-policy global_policy global

Performing a packet-tracer 172.21.2.24 to 172.21.1.251 results in nat-xlate-failed see packet-tracer log

01/pri/act# packet-tracer in PSN-3 icmp 172.21.2.24 0 8 172.21.1.251 det

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f52285835a0, priority=1, domain=permit, deny=false
hits=40031, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=PSN-3, output_ifc=any

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 172.21.1.251 using egress ifc PSN-2

Phase: 3
Type: ACCESS-LIST

Subtype: log
Result: ALLOW
Config:
access-group PSN-3_access_in in interface PSN-3
access-list PSN-3_access_in extended permit object-group DM_INLINE_SERVICE_10 any any
object-group service DM_INLINE_SERVICE_10
service-object ip
service-object icmp
service-object udp
service-object tcp
service-object tcp destination eq domain
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f5228b5feb0, priority=13, domain=permit, deny=false
hits=990, user_data=0x7f521c531500, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 4
Type: NAT
Subtype:
Result: ALLOW
Config:

nat (PSN-3,PSN-2) source dynamic any interface
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f522836f340, priority=6, domain=nat, deny=false
hits=11, user_data=0x7f522836b9d0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=PSN-2

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f52279b8aa0, priority=0, domain=nat-per-session, deny=true
hits=979132, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 6
Type: IP-OPTIONS

Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f522858a170, priority=0, domain=inspect-ip-options, deny=true
hits=2426, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Result:
input-interface: PSN-3
input-status: up
input-line-status: up
output-interface: PSN-2
output-status: up
output-line-status: up
Action: drop
Drop-reason: (nat-xlate-failed) NAT failed

outside

- security 0

inside - 10.10.x.0/24

- security 100

DMZ - 172.16.0.0/24

- security 50

PSN1 - 172.21.0.0/24

- security 70

PSN2 - 172.21.1.0/24

- security 70

PSN3 - 172.21.2.0/24

- security 70

PSN4 - 192.168.11.0/24

- security 70

[rodito]

Is there anyway I can bypass these last two lookups in packet-tracer? Do i need to add no-proxy-arp route-lookup on the nat (PSN*,PSN*) ?

show nat

nat (DMZ,inside) source dynamic any interface
nat (PSN-1,inside) source dynamic any interface
nat (PSN-1,DMZ) source dynamic any interface
nat (PSN-1,PSN-2) source dynamic any interface
nat (PSN-1,PSN-3) source dynamic any interface
nat (PSN-1,PSN-4) source dynamic any interface
nat (PSN-2,inside) source dynamic any interface
nat (PSN-2,DMZ) source dynamic any interface
nat (PSN-2,PSN-1) source dynamic any interface
nat (PSN-2,PSN-3) source dynamic any interface
nat (PSN-2,PSN-4) source dynamic any interface
nat (PSN-3,inside) source dynamic any interface
nat (PSN-3,DMZ) source dynamic any interface
nat (PSN-3,PSN-1) source dynamic any interface
nat (PSN-3,PSN-2) source dynamic any interface
nat (PSN-3,PSN-4) source dynamic any interface
nat (PSN-4,inside) source dynamic any interface
nat (PSN-4,DMZ) source dynamic any interface
nat (PSN-4,PSN-1) source dynamic any interface
nat (PSN-4,PSN-2) source dynamic any interface
nat (PSN-4,PSN-3) source dynamic any interface
nat (DMZ,PSN-1) source dynamic any interface
nat (DMZ,PSN-2) source dynamic any interface
nat (DMZ,PSN-3) source dynamic any interface
nat (DMZ,PSN-4) source dynamic any interface
nat (inside,PSN-1) source dynamic any interface
nat (inside,PSN-2) source dynamic any interface
nat (inside,PSN-3) source dynamic any interface
nat (inside,PSN-4) source dynamic any interface
nat (inside,DMZ) source dynamic any interface

nat (PSN-1,outside) source static any any destination static AC_VPN_pool AC_VPN_pool no-proxy-arp route-lookup
nat (PSN-2,outside) source static any any destination static AC_VPN_pool AC_VPN_pool no-proxy-arp route-lookup
nat (PSN-3,outside) source static any any destination static AC_VPN_pool AC_VPN_pool no-proxy-arp route-lookup
nat (PSN-4,outside) source static any any destination static AC_VPN_pool AC_VPN_pool no-proxy-arp route-lookup
nat (inside,outside) source static any any destination static AC_VPN_pool AC_VPN_pool no-proxy-arp route-lookup
nat (DMZ,outside) source static dmz-subnet dmz-subnet destination static AC_VPN_pool AC_VPN_pool no-proxy-arp route-lookup
nat (outside,outside) source dynamic AnyConnectPool interface

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f52279b8aa0, priority=0, domain=nat-per-session, deny=true
hits=979132, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 6
Type: IP-OPTIONS

Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f522858a170, priority=0, domain=inspect-ip-options, deny=true
hits=2426, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

[Sheraz.Salim]

try this

 no same-security-traffic permit intra-interface
 nat (PSN-3,PSN-2) source dynamic PSN-3 interface

[rodito]

Thanks for the reply. When i performed those it will not accept the PSN-3 after dynamic. I can only input nat (PSN-3,PSN-2) source dynamic any interface
Still can't ping or reach PSN-2

[Sheraz.Salim]

can you run these commands

capture asp type asp-drop all

and


capture capi interface PSN-3 match ip host X.X.X.X host X.X.X.X

Then start ping from the host machine Y.Y.Y.Y and run :
"show cap asp | in Y.Y.Y.Y " to confirm if any packets are getting dropped on ASA.


Along with this, share the output of:
packet-tracer input PSN-3 icmp X.X.X.X 8 0 Y.Y.Y.Y de

[rodito]

fw01/pri/act# show cap asp | in 172.21.2.24
216: 15:44:34.588851 802.1Q vlan#30 P0 172.21.2.24.63988 > 255.255.255.255.1947: udp 40 Drop-reason: (acl-drop) Flow is denied by configured rule
290: 15:44:38.613310 802.1Q vlan#30 P0 172.21.2.24.63988 > 172.21.255.255.1947: udp 40 Drop-reason: (sp-security-failed) Slowpath security checks failed
657: 15:45:14.637875 802.1Q vlan#30 P0 172.21.2.24.63988 > 255.255.255.255.1947: udp 40 Drop-reason: (acl-drop) Flow is denied by configured rule
704: 15:45:18.662349 802.1Q vlan#30 P0 172.21.2.24.63988 > 172.21.255.255.1947: udp 40 Drop-reason: (sp-security-failed) Slowpath security checks failed

fw01/pri/act# packet-tracer input PSN-3 icmp 172.21.2.24 8 0 172.21.1.251 d$

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a0caffc30, priority=13, domain=capture, deny=false
hits=5603, user_data=0x7f4a3286f540, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=PSN-3, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a31d57230, priority=1, domain=permit, deny=false
hits=448292, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=PSN-3, output_ifc=any

Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 172.21.1.251 using egress ifc PSN-2

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group PSN-3_access_in in interface PSN-3
access-list PSN-3_access_in extended permit object-group DM_INLINE_SERVICE_10 any any
object-group service DM_INLINE_SERVICE_10
service-object ip
service-object icmp
service-object udp
service-object tcp
service-object tcp destination eq domain

Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a327fedf0, priority=13, domain=permit, deny=false
hits=2445, user_data=0x7f4a25933540, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (PSN-3,PSN-2) source dynamic any interface
Additional Information:
Dynamic translate 172.21.2.24/0 to 172.21.1.1/15179
Forward Flow based lookup yields rule:
in id=0x7f4a14556d70, priority=6, domain=nat, deny=false
hits=1, user_data=0x7f4a31df9700, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=PSN-2

Phase: 6

Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a30db8a70, priority=0, domain=nat-per-session, deny=true
hits=341296, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a31dcced0, priority=0, domain=inspect-ip-options, deny=true
hits=2480, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 8
Type: SFR
Subtype:
Result: ALLOW
Config:
class-map sfr-class
match any
policy-map global_policy
class sfr-class
sfr fail-open
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a32dccb10, priority=71, domain=sfr, deny=false
hits=2449, user_data=0x7f4a32f9a030, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:

class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a32f97bf0, priority=70, domain=inspect-icmp, deny=false
hits=103, user_data=0x7f4a32dab630, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 10
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a31dcc6e0, priority=66, domain=inspect-icmp-error, deny=false
hits=103, user_data=0x7f4a31d72f40, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any

dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 11
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a0d704bf0, priority=13, domain=capture, deny=false
hits=2, user_data=0x7f4a3286f540, cs_id=0x7f4a31ee1b90, reverse, flags=0x0, protocol=0
src ip/id=172.21.2.24, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.21.1.251, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 12
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (PSN-2,PSN-3) source dynamic any interface
Additional Information:
Forward Flow based lookup yields rule:

out id=0x7f4a31e47820, priority=6, domain=nat-reverse, deny=false
hits=1, user_data=0x7f4a31e42cf0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=PSN-2

Phase: 13
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f4a32fa00c0, priority=0, domain=user-statistics, deny=false
hits=11194, user_data=0x7f4a31bc8a70, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=PSN-2

Phase: 14
Type: NAT
Subtype: per-session
Result: ALLOW
Config:

Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f4a30db8a70, priority=0, domain=nat-per-session, deny=true
hits=341298, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 15
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f4a32da6950, priority=0, domain=inspect-ip-options, deny=true
hits=10706, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-2, output_ifc=any

Phase: 16
Type: CAPTURE
Subtype:

Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x7f4a0e373000, priority=13, domain=capture, deny=false
hits=1, user_data=0x7f4a3286f540, cs_id=0x7f4a31ee1b90, reverse, flags=0x0, protocol=0
src ip/id=172.21.1.251, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.21.2.24, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=PSN-3

Phase: 17
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x7f4a32cf8890, priority=0, domain=user-statistics, deny=false
hits=2484, user_data=0x7f4a31bc8a70, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=PSN-3

Phase: 18

Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 470548, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_sfr
snp_fp_inspect_icmp
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_translate
snp_sfr
snp_fp_inspect_icmp
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop

snp_ifc_stat

Result:
input-interface: PSN-3
input-status: up
input-line-status: up
output-interface: PSN-2
output-status: up
output-line-status: up
Action: allow

fw01/pri/act#

[Sheraz.Salim]

packet-tracer input PSN-3 icmp 172.21.2.24 8 0 172.21.1.251 show working.

[rodito]

That is very weird, I cant seem to get any ping reply from the server from my windows VM inside 172.21.2.24

I also checked all the ports the switch/servers are connected too they are on the right ports in the switch with proper access and trunk and vlan :(

C:\Users\xxx\Downloads>ping 172.21.1.251 -t

Pinging 172.21.1.251 with 32 bytes of data:
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.
Reply from 172.21.2.24: Destination host unreachable.

C:\Users\xxx\Downloads>paping -p 53 172.21.1.251
paping v1.5.5 - Copyright (c) 2011 Mike Lovell

Connecting to 172.21.1.251 on TCP 53:

Connection timed out
Connection timed out
Connection timed out

Connection statistics:
Attempted = 3, Connected = 0, Failed = 3 (100.00%)
Approximate connection times:
Minimum = 0.00ms, Maximum = 0.00ms, Average = 0.00ms

If I try to ping the 10.10.135.x DNS server it reply's just fine.

C:\Users\xxx\Downloads>paping -p 53 10.10.135.49
paping v1.5.5 - Copyright (c) 2011 Mike Lovell

Connecting to 10.10.135.49 on TCP 53:

Connected to 10.10.135.49: time=0.00ms protocol=TCP port=53
Connected to 10.10.135.49: time=0.00ms protocol=TCP port=53
Connected to 10.10.135.49: time=0.00ms protocol=TCP port=53
Connected to 10.10.135.49: time=0.00ms protocol=TCP port=53

Connection statistics:
Attempted = 4, Connected = 4, Failed = 0 (0.00%)
Approximate connection times:
Minimum = 0.00ms, Maximum = 0.00ms, Average = 0.00ms

[Sheraz.Salim]

could be window firewall is on and blocking pings?
can you do
packet-tracer input PSN-3 tcp 172.21.2.24 12345 0 172.21.1.251 80 detail

[rodito]

Highly unlikely I can ping 172.21.1.215 on the ASA VPN and from inside/dmz network just fine.

Windows 7 firewall is completely off on 172.21.2.24

packet-tracer input PSN-3 tcp 172.21.2.24 12345 0 172.21.1.251 80 detail

-- This will not accept the 0 so I removed it.

acdfw01/pri/act# packet-tracer input PSN-3 tcp 172.21.2.24 12345 172.21.1.251 $

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a0caffc30, priority=13, domain=capture, deny=false
hits=412295, user_data=0x7f4a3286f540, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=PSN-3, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a31d57230, priority=1, domain=permit, deny=false
hits=649562, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000

input_ifc=PSN-3, output_ifc=any

Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 172.21.1.251 using egress ifc PSN-2

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group PSN-3_access_in in interface PSN-3
access-list PSN-3_access_in extended permit object-group DM_INLINE_SERVICE_10 any any
object-group service DM_INLINE_SERVICE_10
service-object ip
service-object icmp
service-object udp
service-object tcp
service-object tcp destination eq domain
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a327fedf0, priority=13, domain=permit, deny=false
hits=2598, user_data=0x7f4a25933540, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (PSN-3,PSN-2) source dynamic any interface
Additional Information:
Dynamic translate 172.21.2.24/12345 to 172.21.1.1/12345
Forward Flow based lookup yields rule:
in id=0x7f4a14556d70, priority=6, domain=nat, deny=false
hits=5, user_data=0x7f4a31df9700, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=PSN-2

Phase: 6

Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a30db5370, priority=0, domain=nat-per-session, deny=false
hits=582955, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a31dcced0, priority=0, domain=inspect-ip-options, deny=true
hits=2633, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 8
Type: SFR
Subtype:
Result: ALLOW
Config:
class-map sfr-class
match any
policy-map global_policy
class sfr-class
sfr fail-open
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a32dccb10, priority=71, domain=sfr, deny=false
hits=2595, user_data=0x7f4a32f9a030, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 9
Type: FOVER
Subtype: standby-update
Result: ALLOW

Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a32778a80, priority=21, domain=lu, deny=true
hits=627, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=80, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 10
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f4a0d704bf0, priority=13, domain=capture, deny=false
hits=8, user_data=0x7f4a3286f540, cs_id=0x7f4a31ee1b90, reverse, flags=0x0, protocol=0
src ip/id=172.21.2.24, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.21.1.251, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=any

Phase: 11
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (PSN-2,PSN-3) source dynamic any interface
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f4a31e47820, priority=6, domain=nat-reverse, deny=false
hits=3, user_data=0x7f4a31e42cf0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-3, output_ifc=PSN-2

Phase: 12
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f4a32fa00c0, priority=0, domain=user-statistics, deny=false
hits=11888, user_data=0x7f4a31bc8a70, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=PSN-2

Phase: 13
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f4a30db5370, priority=0, domain=nat-per-session, deny=false
hits=582957, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 14
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f4a32da6950, priority=0, domain=inspect-ip-options, deny=true
hits=11392, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PSN-2, output_ifc=any

Phase: 15
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x7f4a0e373000, priority=13, domain=capture, deny=false
hits=2, user_data=0x7f4a3286f540, cs_id=0x7f4a31ee1b90, reverse, flags=0x0, protocol=0
src ip/id=172.21.1.251, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.21.2.24, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=PSN-3

Phase: 16
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x7f4a32cf8890, priority=0, domain=user-statistics, deny=false

hits=2629, user_data=0x7f4a31bc8a70, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=PSN-3

Phase: 17
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 497400, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_sfr
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_sfr
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Result:
input-interface: PSN-3
input-status: up
input-line-status: up
output-interface: PSN-2
output-status: up
output-line-status: up
Action: allow

[Sheraz.Salim]

This is not firewall issue. as ping and port 80 showing working.

what is the setup like from the firewall point of view. switching is accurate is working ok?

[rodito]

The ASA connects to a Cisco switch which I have setup the trunking for the connection to the ASA-Switch. The rest is configured as per VLAN access port. I'm sure its configured right unless one of my admins put the wrong cable on the wrong port. I know they installed a new server and they plug the Ethernet to that switch without talking to me. So that is the only change on the switch side -- they were saying they plug it in to the right access port but only way I can check is when i connect to the USB console. The switch will not allow me to login remotely via ssh for some reason admin doesn't work with ssh but the web interface works fine i can see the connected ports and type.

Only time I can check is Friday unfortunately I work from home and that the only time I can go to the DC to check the cables myself.

[Sheraz.Salim]

can you ping the server from firewall?

if so

show arp | i X.X.X.X      (YOUR SERVER IP)

this above command will give you the mac address of your server.

also can you set a capture on firewall while you pining the server from your windows 7. as packet trace is saying it working. just to confirm we are right that firewall is working but issue is somewhere at switch side.

[rodito]

fw01/pri/act# show cap capi | in 172.21.2.24
1: 15:46:31.471487 802.1Q vlan#30 P0 172.21.2.24 > 172.21.1.251: icmp: echo request
2: 15:53:14.257402 802.1Q vlan#30 P0 172.21.2.24 > 172.21.1.251: ip-proto-0, length 16
3: 15:53:39.505543 802.1Q vlan#30 P0 172.21.2.24 > 172.21.1.251: icmp: echo reply
4: 15:54:04.273011 802.1Q vlan#30 P0 172.21.2.24 > 172.21.1.251: icmp: echo request
5: 16:06:10.636853 802.1Q vlan#30 P0 172.21.2.24.12345 > 172.21.1.251.80: S 1672473560:1672473560(0) win 8192
6: 16:09:32.805424 802.1Q vlan#30 P0 172.21.2.24 > 172.21.1.251: icmp: echo request
fw01/pri/act#

fw01/pri/act# show arp | i 172.21.1.251
PSN-2 172.21.1.251 0050.5688.1082 2
fw01/pri/act#

fw01/pri/act# ping
TCP Ping [n]:
Interface: PSN-2
Target IP address: 172.21.1.251
Repeat count: [5]
Datagram size: [100]
Timeout in seconds: [2]
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.21.1.251, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
fw01/pri/act#

[Sheraz.Salim]

could you figured out what was the issue. as packet trace was was patch is working. any chance if you look at the switch side to spot any issue.