Solved: Adding another IPSec tunnel

Collin Clark · ‎12-26-2010

I have two remote sites (PIX501's) connecting to the hub (2811). All sites can talk to all sites. For the spokes to talk, they go through the hub. I'm trying add a third site and I can get connectivity to the hub no problem, but not the spokes. Below is a show crypto ipsec sa from the new site.

Hub site: 192.168.12.0 /24

Remote site A: 192.168.13.0 /24 (working)

Remote site B: 192.168.14.0 /24 (working)

Remote site C: 192.168.15.0 /24 (not working)

There is also a diagram attached that show all this a little better.

This looks good other than there are no remotes sites. Connectivity to this network is just fine and it is the hub.

access-list 130 extended permit ip 192.168.15.0 255.255.255.0 192.168.12.0 255.255.255.0
local ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.12.0/255.255.255.0/0/0)

Here is the same command from one of the working sites.

local ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.12.0/255.255.255.0/0/0)

local ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)

local ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.14.0/255.255.255.0/0/0)

As you can see it builds the tunnel to the hub and each remote site. I can't get the new ASA to do the same. It looks to me like the problem is on the hub. I've removed the crypto map from the interface, built the VPN config multiple times, etc, etc. Please tell me I'm missing something easy. Thanks.

Marcin Latosiewicz · ‎12-29-2010

Collin,

Can you attach full IPSec SA output? Not only the heads from both spoke and hub.

If you add "set reverse static" on hub side routes will be added regardless of IPSec SAs being up.

I understand you already made sure traffic between 192.168.13.0/24 and 192.168.15.0/24 (and vice versa) are not being NATed.

Marcin

View solution in original post

Marcin Latosiewicz · ‎12-26-2010

Collin,

local ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)

Is that tunnel up? Does it have valid SPIs? Does it pass traffic in any direction?

"show crypto ipsec sa | i caps|ident|spi"

If there is a tunnel to remote C for subnets in remove site A.

There should be also a tunnel from to remote site A for subnets from site C.

In any way let's check if that particular SA is up and passing any traffic.

Marcin

Collin Clark · ‎12-28-2010

Marcin,

Thanks for taking the tim to look at this. Traffic was sent from 192.168.13.0 to 192.168.15.0 and it was one way traffic as expected.

   local ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)
    #pkts encaps: 29, #pkts encrypt: 29, #pkts digest 29
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0

Again I apprecieate your time, but I'm sick of working on this and I am going to open a TAC case.

Marcin Latosiewicz · ‎12-28-2010

Collin,

That's probably for the best ;-)

I guess you will need to investigate why traffic from other side is not hittin crypto, possibly a NAT or routing problem on the remote end?

Marcin

Collin Clark · ‎12-29-2010

I found that the IPSec SA's are not built until traffic is sent across them. Anyway, the IPSec SA's show encryption on the new site and decryption on the hub, but I never get to the other sites. Looks like it could be a routing problem on the hub. Any suggestions? I tried opening a TAC case but Cisco is giving me the run around on associating my ID with the service contracts. I really appreciate your help.

Marcin Latosiewicz · ‎12-29-2010

Collin,

Let's give it a try!

The SA you mentioned before:

   local  ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)
    #pkts encaps: 29, #pkts encrypt: 29, #pkts digest 29
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0

where was it taken from?

It shows us being able to encapsulate packets and most likely decapsulating them on the other side.

You should check the peer and see if you're reciving decapsulations there.

Marcin

Collin Clark · ‎12-29-2010

That was taken from one of the other remote networks. I see it get encaps, but no decaps. I just checked and at the hub I don't have an IPSec SA for the hub to that particular remote site (192.168.13.0). Let me get that working and see if that's the issue. Thanks again.

Marcin Latosiewicz · ‎12-29-2010

Collin,

I guess you do have the access list defined, maybe problem is with proxy IDs?

Check those debugs:

debug crypt isa 127

debug crypto ipsec 127

while you're pushhing traffic through to the other peer.

If you're getting too mich info make the debug conditional:

debug crypto condition peer ...

Just make sure that you do this while trying to bring up the SA :-)

BTW, if you want on-all-the-time tunnel best use GRE or VTI (neither available on PIX/ASA tho).

Marcin

Collin Clark · ‎12-29-2010

I ran the debug on 192.168.13.0 PIX and it doesn't show IPSec SA's being built for 192.168.15.0. As you know, with a PIX I have no way of generating traffic. I'll see if I can reach out to them.

Collin Clark · ‎12-29-2010

I have built the tunnel. Maybe I'm getting things screwed up (been working on this way too long). I'm going to write out what is working where. This is for my etification and sanity. I have a tunnel built from 192.168.15.0 to 192.168.13.0 but it goes through the hub which is 192.168.12.0.

192.168.15.0 to 192.168.13.0 from the ASA at 192.168.15.0-

local ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
      #pkts encaps: 9, #pkts encrypt: 9, #pkts digest: 9
      #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

Same tunnel for the hub side-

local ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 9, #pkts decrypt: 9, #pkts verify: 9
     current outbound spi: 0xE162B7A7(3781343143)
      spi: 0x66CB9241(1724617281)
      spi: 0xE162B7A7(3781343143)

Looks to me that is working correctly. Now lets go to the remote site (192.168.13.0)-

local ident (addr/mask/prot/port): (192.168.13.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (192.168.15.0/255.255.255.0/0/0)
    #pkts encaps: 29, #pkts encrypt: 29, #pkts digest 29
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0
     current outbound spi: 0

OK it's never getting decrypted here. Is the flow correct here?

Marcin Latosiewicz · ‎12-29-2010

Collin,

The remote side 192.168.13.0 should show also decapsulations.

BUT I noticed it has oupbound SPI of 0? Meaning this tunnel is not up.

The bottom line is

When going through tunnels you need to have following IPSec SAs.

on 192.168.15.0 side

1) from 192.168.15.0 (local) to 192.168.13.0 (remote) go to HUB.

on hub side.

2) from 192.168.13.0 (local) to 192.168.15.0 (remote) to go peer ... (public IP of 192.168.15.0)

3) from 192.168.15.0 (local) to 192.168.13.0 (remote) to go peer ...(public IP of site which has 192.168.13.0)

on 192.168.13.0 side:

4) from 192.168.13.0 (local) to 192.168.15.0 (remote) to go to HUB.

So a simple ICMP echo request would do:

Encaps on 1) decaps on 2) encaps on 3) decaps on 4).

ICMP echo reply would do:

Encaps on 4) decaps on 3) encaps on 2) decaps on 1)

(I might have screwed something up, I'm writing without verification).

What you've shown me is that IPSec SA 2) is up. But is IPSec SA 3) and 4) up?

Marcin

Collin Clark · ‎12-29-2010

I agree that the SPI of 0 is strange. All of my tunnels show 0 except for local 192.168.13.0 to the hubs directly connected 192.168.12.0. I've attached a diagram with the IPSec SA's for each site. If I'm doing something wrong, this will hopefully help you help me :-)

Marcin Latosiewicz · ‎12-29-2010

Collin,

Something's off here.

You're seeing a LOT more encaps and decaps on hub side than on either of the spokes.

What I don't quite get is that there is a phase 2 on hub for 192.168.13.0 but not on the spokes itself I would assume it's part of the problem. And would indicate a failed Quick mode.

As a first steps check if both sides have correct (mirrored) access-lists. But I guess a debug might be needed.

Are you using RRI?

I noticed the 4 packets send from 192.168.15.0 and received by hub, but those packets don't seem to be hitting encapsulations on the hub side towards 192.168.13.0, can you check routing?

You might add "set reverse-route static" under crypto map entries on on hub side.

For debugging,

debug crypto condition peer ipv4 (public_IP) on router

debug cry isa

debug crypto ipsec

on both router and PIX.

Let's see why they are failing.

Marcin

Collin Clark · ‎12-29-2010

I am using RRI. I beleive this is a routing issue on the hub, but I don't know what to add. I tried adding the remote site like this-

On the hub-

ip route 192.168.13.0 255.255.255.0 s0/0/0

I also entered on the remotes, but nothing changed. My other spokes communicate fine without routing which confuses me even more.

Marcin Latosiewicz · ‎12-29-2010

Collin,

Can you attach full IPSec SA output? Not only the heads from both spoke and hub.

If you add "set reverse static" on hub side routes will be added regardless of IPSec SAs being up.

I understand you already made sure traffic between 192.168.13.0/24 and 192.168.15.0/24 (and vice versa) are not being NATed.

Marcin