Getting intermittent connection to internal resources from vpn remote connection

[iglablues]

Hi there,

I have an intermittent issue happening on my company's firewall, and I'm at a loss as to how to troubleshoot further. I'm hoping someone can give me some tips or pointers in the right direction.

We recently made some changes to our network, which included moving an ASA 5505 from one location to another. In moving this we also connected it to a different switch. It used to hang off a Cisco 3560, and now it hangs off a ProCurve 5406zl.

The setup is that the ASA is connected on two ports to the HP, which is simply a layer 2 device sitting between the firewall and the uplink to our ISP. One port, e0/0, is the outside interface and is set to switchport access vlan 2. The second interface, e0/1, is set for vlan 1, also access mode. It has an IP of 10.0.0.2. The HP on the other end of that has an IP of 10.0.0.1. So outside VPN connections come through the HP to the ASA on e0/0, and back out the ASA on port e0/1 to reach devices on the internal network.

What happens is that when a user connects to the VPN, they can reach internal resources intermittently. For example, I tried to ping an internal server IP address (let's say 10.0.0.23) from my laptop, while on the VPN, and pings failed. However I could ping that IP from the ASA itself. Another example is that during one VPN connection I was unable to connect to an internal web server, but once I disconnected and connected ahain I could reach the server fine.

The intermittent nature of the problem made me think that it could be an ARP issue, that somehow the traffic is getting sent back out the wrong interface sometimes, hence the lack of communication. However when I did show switch mac-address | include mac address, using the base mac of the HP, I only saw the mac address of the HP on one interface, e0/1. That makes me think that things are working as they should. Although I guess maybe it should appear on both interfaces...? I don't know. Can anyone share some ideas of what the problem might be, or how I can most effectively troubleshoot this? It seems like the symptoms are indicative of some kind of rookie mistake, but for the life of me I can't figure out what it is.

I appreciate your responses. Thanks.

[nine_2012]

the time you are not able to connect to internal resources, did you try clearing the arp table at the switch end and check if you can reach internal resources?

[iglablues]

Hi Nitin,

Thanks for the suggestion. I no longer think it's an ARP issue after observing the behavior more. Pretty frequently what happens is that upon establishing a vpn session I am unable to ping devices within the internal network, all private IPs. I disconnect from vpn and try again, and on the second go around those connections suddenly work. It seems to be something odd with the tunnel itself, but this is beyond my skillset.

There are quite a few views of this question but disappointingly little response. I'm not asking for anyone to fix this for me, but rather to point me in the right direction, i.e. what kind of commands can I run on the ASA, what sort of things could cause this behavior. Again, I appreciate input.

Thanks.

[Jeff Van Houten]

You said you moved the Asa. When you did, did you change any of the subnets published to the client to route over the VPN?

Sent from Cisco Technical Support iPad App

[iglablues]

Hi Jeff, and thanks. I didn't change anything in the config actually. The IP ranges didn't change, nor did the IP of the ASA itself. In its old setting it hung off of an intermittent Cisco switch, let's say IP 10.100.0.25. There were two other switches that used VRRP with a shared IP of 10.100.10.1 and acted as the default gateway for all devices on the network, except the ASA. The gateway switches in turn used the ASA, 10.100.10.2, as the default gateway. So, internal traffic would flow through 10.100.0.1 unless it needed to access an external network, i.e. the WAN, in which case the switch at 10.100.10.1 would send the traffic to the ASA at 10.100.10.2. The ASA's default gateway was the public IP of our ISP.

In the change we moved from using a Cisco core to an HP core, so the ASA was moved to hang directly off the HP. The HP was re-IP'd as 10.100.10.1, the ASA stayed the same. I didn't change any configuration on the ASA, and the only thing did to the HP was to change it's default gateway to point to the ASA (same as the old setup), and set the ports for the different vlans.

What I've also found is that at times once I establish a vpn connection I cannot even ping the connected HP switch from my client machine, but I can ping it from the ASA. If I disconnect from the vpn and try again, it may work.  Here's some output based on commands in this link,

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a00807c35e7.shtml#s1. For this command I am attempting to first ping 10.100.10.1 (the HP switch) and then ssh to it.

asa# capture capin interface inside match ip host 10.100.15.3 host 10.100$

asa# show cap

asa# show capture capin

18 packets captured

   1: 11:00:24.574997 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

   2: 11:00:24.577210 802.1Q vlan#1 P6 10.100.10.1 > 10.100.15.3: icmp: echo reply

   3: 11:00:25.585342 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

   4: 11:00:26.576676 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

   5: 11:00:27.584259 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

   6: 11:00:28.577896 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

   7: 11:00:29.590682 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

   8: 11:00:30.579529 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

   9: 11:00:31.588363 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

  10: 11:00:32.588317 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request

  11: 11:00:38.699441 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

  12: 11:00:39.642651 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

  13: 11:00:40.645352 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

  14: 11:00:41.648815 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

  15: 11:00:42.649777 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

  16: 11:00:43.659832 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

  17: 11:00:45.654049 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

  18: 11:00:49.664699 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S 2589950443:2589950443(0) win 65535

18 packets shown

asa# show capture capin detail

18 packets captured

   1: 11:00:24.574997 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 38278)

   2: 11:00:24.577210 441e.a16a.8b00 001d.45e5.85ce 0x8100 102: 802.1Q vlan#1 P6 10.100.10.1 > 10.100.15.3: icmp: echo reply [tos 0xe0]  (ttl 255, id

   3: 11:00:25.585342 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 9469)

   4: 11:00:26.576676 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 50837)

   5: 11:00:27.584259 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 24048)

   6: 11:00:28.577896 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 57505)

   7: 11:00:29.590682 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 8438)

   8: 11:00:30.579529 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 64136)

   9: 11:00:31.588363 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 24534)

  10: 11:00:32.588317 001d.45e5.85ce 441e.a16a.8b00 0x8100 102: 802.1Q vlan#1 P0 10.100.15.3 > 10.100.10.1: icmp: echo request (ttl 64, id 56295)

  11: 11:00:38.699441 001d.45e5.85ce 441e.a16a.8b00 0x8100 82: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

  12: 11:00:39.642651 001d.45e5.85ce 441e.a16a.8b00 0x8100 82: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

  13: 11:00:40.645352 001d.45e5.85ce 441e.a16a.8b00 0x8100 82: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

  14: 11:00:41.648815 001d.45e5.85ce 441e.a16a.8b00 0x8100 82: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

  15: 11:00:42.649777 001d.45e5.85ce 441e.a16a.8b00 0x8100 82: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

  16: 11:00:43.659832 001d.45e5.85ce 441e.a16a.8b00 0x8100 82: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

  17: 11:00:45.654049 001d.45e5.85ce 441e.a16a.8b00 0x8100 82: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

  18: 11:00:49.664699 001d.45e5.85ce 441e.a16a.8b00 0x8100 66: 802.1Q vlan#1 P0 10.100.15.3.57943 > 10.100.10.1.22: S [tcp sum ok] 2589950443:2589950

18 packets shown

asa# show capture capin trace                           

7 packets captured

   1: 11:04:53.510685 802.1Q vlan#1 P0 10.100.15.3.57954 > 10.100.10.1.22: S 1151368455:1151368455(0) win 65535

   2: 11:04:54.491414 802.1Q vlan#1 P0 10.100.15.3.57954 > 10.100.10.1.22: S 1151368455:1151368455(0) win 65535

   3: 11:04:55.482488 802.1Q vlan#1 P0 10.100.15.3.57954 > 10.100.10.1.22: S 1151368455:1151368455(0) win 65535

   4: 11:04:56.499119 802.1Q vlan#1 P0 10.100.15.3.57954 > 10.100.10.1.22: S 1151368455:1151368455(0) win 65535

   5: 11:04:57.489308 802.1Q vlan#1 P0 10.100.15.3.57954 > 10.100.10.1.22: S 1151368455:1151368455(0) win 65535

   6: 11:04:58.493275 802.1Q vlan#1 P0 10.100.15.3.57954 > 10.100.10.1.22: S 1151368455:1151368455(0) win 65535

   7: 11:05:00.495473 802.1Q vlan#1 P0 10.100.15.3.57954 > 10.100.10.1.22: S 1151368455:1151368455(0) win 65535

7 packets shown

Here are some additional potentially relevant sections of my ASA config (public IPs changed for security):

asa# show route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP

       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

       * - candidate default, U - per-user static route, o - ODR

       P - periodic downloaded static route

Gateway of last resort is 1.1.1.125 to network 0.0.0.0

C    1.1.1.64 255.255.255.192 is directly connected, outside

S    10.100.15.3 255.255.255.255 [1/0] via 1.1.1.125, outside

S    10.105.0.0 255.255.0.0 [1/0] via 10.100.10.1, inside

C    10.100.0.0 255.255.0.0 is directly connected, inside

S*   0.0.0.0 0.0.0.0 [255/0] via 1.1.1.125, outside

Thanks in advance for your suggestions and observations.