Cisco ASA 5505 LAN Question

[jkelnhofer]

So I have run into an issue I have never seen before and I thought I would share the issue with a group.

We installed a new ASA 5505 at a site using 3 interfaces:

E0/0 = WAN

E0/1 = 192.168.1.1

E0/2 = 192.168.2.1

We setup interVLAN routing between E0/1 and E0/2 and both VLANs go out the WAN interface for internet.  All is working great except for one thing.  I'll do my best to explain the issue.

So this site has 5 Point of Sale stations and a POS server all statically assigned and segmented on their own switch with a connection to the ASA.

POS1 = 192.168.2.11

POS2 - 192.168.2.102

POS3 - 192.168.2.103

POS4 - 192.168.2.104

POS5 - 192.168.2.5

POS Server Appliance - 192.168.2.250

The POS1 Station runs an app called NETePay (192.168.2.11).  The other 4 POS stations connect through this device via a software application.  I have no software firewalls enabled or AV running (for testing purposes).  The NETePay application is what processes the Credit Card transactions and on POS1 they work flawlessly.  All stations can talk to the POS server and all have internet access and such.  The problem resides when the POS2-5 stations try to talk to the POS1 device to process the transactions.  They all fail.

HOWEVER, if we run a continuous pings on the POS2-5 devices (i.e. "ping 192.168.2.11 -t") the transactions go through with no issues on all devices.  If the pings are NOT running the transactions fail. 

My question is:  What in the ASA would cause traffic to NOT route vs be ABLE to route when a continuous PING is running?  Could this be a NAT issue with the VLAN itself?  I can ping every station, connect to the POS Server, run all day long with no issues, but the connection to 192.168.2.11 for CC processing only works if I am running a continuous ping.  If it was a port blocking issue, I would assume it would not work at all.  I have also taken the following troubleshooting steps with no luck toward resolution:

1.  Cleared ARP tables on ASA and all POS Devices

2.  Replaced the Switch with a basic layer2 switch

3.  Performed a packet capture from the POS client to the POS Host (192.168.2.5 to 192.168.2.11).  I sent that to the POS vendor and they stated that it said there was a duplicate IP address on the network.  That IP was 192.168.2.250 with the MAC of the ASA and the MAC of the POS server but there is not 2.250 address programmed in the ASA at all. 

4.  Contacted TAC.  They added a "same-security-traffic permit intra-interface" command that seemed to help, but did not resolve the problem.

So I am at a loss here.  I don't claim to be a Cisco Guru by any means, but I do know my way around an ASA and again, have never seen something like this before.  I am new to this forum so I hope I didn't break any rules by asking this question here, but I figured if anyone would know where to start, this would be the place.

Sincerely,

Jason

[GRANT3779]

Interesting issue. I wonder about the pinging from POS2-5 to POS1 as from what I can make out above, this would not involve the ASA at all.

Where does the POS Server come into play in all this.

Can you supply the MACs of each POS machine, also the ouptut of the mac address table of the switch these devices reside on.

Relevant ASA config would be beneficial also.

To confirm, the POS switch hangs of E0/2 ? 

Is the E0/1 interface/network irrelevant in this issue?

[jkelnhofer]

First off, thank you for the reply.  This issue is nothing less than frustrating 

Where does the POS Server come into play in all this.

So the POS server is an appliance from ECRS (POS Vendor) that's in the rack and looks quite like a switch.  It holds all the POS DATA (Pricing, Inventory, sales, and anything else related to a grocery store).  It's a "server" if you want to call it that.  Address = 192.168.2.250

Can you supply the MACs of each POS machine, also the ouptut of the mac address table of the switch these devices reside on.

I can provide these, however I need to get back onsite to get them.  The organization has a very strict "no remote access" policy and being the vendor/partner I have to respect that.  can go onsite and get this information in the near future.

Relevant ASA config would be beneficial also.

ASA Version 9.1(6) 
! 
hostname ciscoasa 
enable password ***************** encrypted 
names 
! 
interface Ethernet0/0 
switchport access vlan 2 
! 
interface Ethernet0/1 
! 
interface Ethernet0/2 
switchport access vlan 12 
! 
interface Ethernet0/3 
! 
interface Ethernet0/4 
! 
interface Ethernet0/5 
! 
interface Ethernet0/6 
! 
interface Ethernet0/7 
! 
interface Vlan1 
description Connection to Internal LAN 
nameif inside 
security-level 100 
ip address 192.168.1.1 255.255.255.0 
! 
interface Vlan2 
description Connection to Internet 
nameif outside 
security-level 0 
ip address dhcp setroute 
! 
interface Vlan12 
nameif POS 
security-level 100 
ip address 192.168.2.1 255.255.255.0 
! 
ftp mode passive 
dns domain-lookup outside 
dns domain-lookup inside 
dns domain-lookup POS 
dns server-group DefaultDNS 
name-server 8.8.8.8 
name-server 8.8.4.4
same-security-traffic permit inter-interface 
same-security-traffic permit intra-interface 
object network obj_any 
subnet 0.0.0.0 255.255.255.0 
object network POS 
subnet 0.0.0.0 255.255.255.0 
object network Internal-LAN 
subnet 192.168.1.0 255.255.255.0 
object network POS-LAN 
subnet 192.168.2.0 255.255.255.0 
description POS connection to internet 
access-list outside_access_in extended permit icmp any any echo-reply 
pager lines 24 
logging asdm informational 
mtu outside 1500 
mtu inside 1500 
mtu POS 1500 
no failover 
icmp unreachable rate-limit 1 burst-size 1 
no asdm history enable 
arp timeout 14400 
no arp permit-nonconnected 
nat (any,any) source static POS POS 
! 
object network obj_any 
nat (inside,outside) dynamic interface 
object network Internal-LAN 
nat (any,outside) dynamic interface 
object network POS-LAN 
nat (any,outside) dynamic interface 
! 
nat (any,any) after-auto source static any any 
access-group outside_access_in in interface outside 
route outside 0.0.0.0 0.0.0.0 10.0.0.1 1 
timeout xlate 3:00:00 
timeout pat-xlate 0:00:30 
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02 
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00 
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00 
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute 
timeout tcp-proxy-reassembly 0:01:00 
timeout floating-conn 0:00:00 
dynamic-access-policy-record DfltAccessPolicy 
user-identity default-domain LOCAL 
http server enable 
http 192.168.1.0 255.255.255.0 inside 
no snmp-server location 
no snmp-server contact 
snmp-server enable traps snmp authentication linkup linkdown coldstart warmstart 
crypto ipsec security-association pmtu-aging infinite 
crypto ca trustpool policy 
telnet timeout 5 
no ssh stricthostkeycheck 
ssh timeout 5 
ssh key-exchange group dh-group1-sha1 
console timeout 0 

dhcpd auto_config outside 
! 
dhcpd address 192.168.1.5-192.168.1.254 inside 
dhcpd dns 8.8.8.8 8.8.4.4 interface inside 
dhcpd enable inside 
! 
dhcpd address 192.168.2.5-192.168.2.100 POS 
dhcpd dns 8.8.8.8 8.8.4.4 interface POS 
dhcpd enable POS 
! 
threat-detection basic-threat 
threat-detection statistics access-list 
no threat-detection statistics tcp-intercept 
! 
class-map inspection_default 
match default-inspection-traffic 
! 
! 
policy-map type inspect dns preset_dns_map 
parameters 
message-length maximum client auto 
message-length maximum 512 
policy-map global_policy 
class inspection_default 
inspect dns preset_dns_map 
inspect ftp 
inspect h323 h225 
inspect h323 ras 
inspect rsh 
inspect rtsp 
inspect esmtp 
inspect sqlnet 
inspect skinny 
inspect sunrpc 
inspect xdmcp 
inspect sip 
inspect netbios 
inspect tftp 
inspect ip-options 
! 
service-policy global_policy global 
prompt hostname context 
call-home reporting anonymous prompt 2 
Cryptochecksum:0e278a11f11f97a44df23f673727dd9b 
: end 

To confirm, the POS switch hangs of E0/2 ? 

That is correct, and the only devices on that switch are POS Stations, the POS Server and the connection to the interface on the ASA

Is the E0/1 interface/network irrelevant in this issue?

Again, correct.  I need some interVLAN routing in order for their wireless Handhelds to be able to reach the POS Server 192.168.2.250 so they can do inventory and price updates out on the floor.  They have plans for a new WiFi system that I can eventually tie to the 192.168.2.0 network, but for now they have to route from the 192.168.1.0 network.  Eventually PCI will come into play and need to be completely segmented, but for the sake of this issue, yes it is irrelevant. 

[GRANT3779]

I have attached quick diagram just to check I am right in thinking how this hangs together.

With regards to all the NAT statements in the config - can you explain the thinking behind each of them and the order they were configured in?

There are only two networks that need to be NAT'd outbound on ASA, is that correct? 192.168.1.0/24  and 192.168.2.0/24

Also, can you provide output of the show nat command on the ASA.