VPN Spoke-Hub-Spoke Troubleshooting

[aw3000]

Hi all,

I am having some problems with a spoke-hub-spoke setup and I'm really stumped.  Hopefully some knowledgeable person can help me out...

I have three sites: a hub and spokes A and B.  Site-to-site VPNs are configured between spoke A and the hub, and spoke B and the hub.  The hub can communicate with each spoke, but they can't communicate with each other.

Things I have already done:

Spoke A ASA: added the spoke B subnets to the interesting traffic and no-NAT

Spoke B ASA: added the spoke A subnets to the interesting traffic and no-NAT

Hub ASA: configured same-security-traffic inter-interface; configured no-NAT from the spoke A subnets to the spoke B subnets on the outside interface

The only routing in place on the hub ASA is 0.0.0.0 to the outside interface.

Using the command show crypto sa peer <peer ip> on the hub ASA, I can see that traffic for spoke B is arriving from spoke A and being de-encapsulated, but not being encapsulated and sent to spoke B.  Same for traffic from spoke B to spoke A.

I will post santised configs if required, but is there anything I could have missed?

Many thanks

Alex

[Rahul Govindan]

Strange indeed. No reason I can see that the ASA should not encrypt. You have the right outside,outside NAT rule to match between SiteA and SiteB. 

nat (outside,outside) source static Site_A_Networks Site_A_Networks destination static Site_B_Networks Site_B_Networks no-proxy-arp route-lookup

Can you get the "show nat detail" to see if the show the order of the NAT statements on the ASA.

Also, do you have other VPN tunnels that you have not pasted in the config? I see sequence number 2 and 6 on your config. Are there other crypto map sequences that you have not shown here, which could conflict with the tunnels shown here?

Last point to make is the version on the hub. The 9.2(2)4 version is very old. There have been many bugs in the earlier 9.x versions regarding Stale VPN contexts, which caused failure in encapsulation. A reboot usually fixes this. Is this something you can try in a window? 

[aw3000]

Hi Rahul,

Result of the command: "sh nat detail"

Manual NAT Policies (Section 1)
1 (outside) to (outside) source static Site_A_Networks Site_A_Networks destination static Site_B_Networks Site_B_Networks no-proxy-arp route-lookup
translate_hits = 1113612, untranslate_hits = 1113618
Source - Origin: 10.25.0.0/16 Translated: 10.25.0.0/16
Destination - Origin: 10.12.0.0/16 Translated: 10.12.0.0/16
2 (dmz) to (outside) source static Hub_DMZ Hub_DMZ destination static Site_A_Networks Site_A_Networks no-proxy-arp route-lookup
translate_hits = 733, untranslate_hits = 6320005
Source - Origin: 10.50.8.0/24, Translated: 10.50.8.0/24
Destination - Origin: 10.25.0.0/16 Translated: 10.25.0.0/16
3 (inside) to (outside) source static Hub_Networks Hub_Networks destination static DM_INLINE_NETWORK_15 DM_INLINE_NETWORK_15 no-proxy-arp route-lookup
translate_hits = 1314878, untranslate_hits = 1328239
Source - Origin: 10.50.0.0/16, Translated: 10.50.0.0/16
Destination - Origin: 10.25.0.0/16, 10.12.0.0/16
Translated: 10.25.0.0/16, 10.12.0.0/16
13 (outside) to (inside) source static Hub_VPN Hub_VPN destination static Hub_Networks Hub_Networks no-proxy-arp
translate_hits = 8837081, untranslate_hits = 9527003
Source - Origin: 10.50.10.0/24, Translated: 10.50.10.0/24
Destination - Origin: 10.50.0.0/16, Translated: 10.50.0.0/16
14 (outside) to (outside) source static Hub_VPN Hub_VPN destination static Site_A_Networks Site_A_Networks no-proxy-arp description
translate_hits = 1290663, untranslate_hits = 1293660
Source - Origin: 10.50.10.0/24, Translated: 10.50.10.0/24
Destination - Origin: 10.25.0.0/16 Translated: 10.25.0.0/16

I have removed a few NAT rules related to specific services running in the DMZ which I don't think would be relevant.

There are indeed some other tunnels to what I could have named Site C and Site D, but I took them out of the sanitised config for the sake of simplicity. I've been concentrating on resolving the issue between two sites as the solution should be valid for the others too.



I think your suggestion of at least rebooting and if possible upgrading IOS is a good one and I will schedule it as soon as I can.



Thanks and regards



Alex

[Rahul Govindan]

Nat Rule#1 seems to show translates and untranslates - which is good:

1 (outside) to (outside) source static Site_A_Networks Site_A_Networks destination static Site_B_Networks Site_B_Networks no-proxy-arp route-lookup
translate_hits = 1113612, untranslate_hits = 1113618
Source - Origin: 10.25.0.0/16 Translated: 10.25.0.0/16
Destination - Origin: 10.12.0.0/16 Translated: 10.12.0.0/16

I see that you have removed the following from Site B networks:

network-object object 10.10.0.0
network-object object 10.13.13.0
network-object object 10.80.0.0

The reason I asked for the other tunnels is to check if there could be a possible conflicts in the crypto ACL. The crypto map is matched in sequence, so if there is an entry that is higher up that matches the traffic, it will try to send it to through that tunnel instead. 

Reload will definitely help if this is a bug. If this is still not resolved after a reload, there is most likely a conflicting entry somewhere. 

[aw3000]

Hi Rahul,



Here's the config again, with the key objects, NAT and all of our crypto maps. I have now included all the sites that we have in reality, only with the external IPs and site names changed. Note that 'Site B' is now called 'US' in order to better reflect reality. I'm pretty sure that there shouldn't be any issue with the order of the crypto maps but it would be great if you could check!



object-group network Hub_Networks

network-object object 10.50.0.0

object-group network Site_A_Networks

network-object object 10.25.0.0

object-group network Site_C_Networks

network-object object 10.20.0.0

object-group network Site_D_Networks

network-object object 10.30.0.0

object-group network EU_Networks

group-object Site_D_Networks

group-object Site_A_Networks

group-object Site_C_Networks

object-group network US_Networks

network-object object 10.12.0.0

network-object object 10.10.0.0

network-object object 10.13.13.0

network-object object 10.80.0.0

object-group network DM_INLINE_NETWORK_13

group-object Hub_Networks

group-object EU_Networks

object-group network DM_INLINE_NETWORK_2

group-object US_Networks

group-object Hub_Networks

object-group network DM_INLINE_NETWORK_15

group-object EU_Networks

group-object US_Networks



access-list outside_cryptomap_Site_C extended permit ip object-group DM_INLINE_NETWORK_2 object-group Site_C_Networks

access-list outside_cryptomap_Site_D extended permit ip object-group DM_INLINE_NETWORK_2 object-group Site_D_Networks

access-list outside_cryptomap_US extended permit ip object-group DM_INLINE_NETWORK_13 object-group US_Networks

access-list outside_cryptomap_Site_A extended permit ip object-group DM_INLINE_NETWORK_2 object-group Site_A_Networks



nat (outside,outside) source static EU_Networks EU_Networks destination static US_Networks US_Networks no-proxy-arp route-lookup

nat (dmz,outside) source static Hub_DMZ Hub_DMZ destination static EU_Networks EU_Networks no-proxy-arp route-lookup

nat (inside,outside) source static Hub_Networks Hub_Networks destination static DM_INLINE_NETWORK_15 DM_INLINE_NETWORK_15 no-proxy-arp route-lookup

nat (outside,dmz) source static Hub_VPN Hub_VPN destination static Hub_DMZ Hub_DMZ no-proxy-arp

nat (outside,inside) source static Hub_VPN Hub_VPN destination static Hub_Networks Hub_Networks no-proxy-arp

nat (outside,outside) source static Hub_VPN Hub_VPN destination static EU_Networks EU_Networks no-proxy-arp

nat (any,outside) source dynamic 10.50.0.0 interface



crypto map outside_map 1 match address outside_cryptomap_Site_C

crypto map outside_map 1 set peer 4.4.4.4

crypto map outside_map 1 set ikev1 transform-set ESP-AES-256-SHA ESP-AES-192-SHA ESP-AES-128-SHA

crypto map outside_map 2 match address outside_cryptomap_Site_A

crypto map outside_map 2 set peer 2.2.2.2

crypto map outside_map 2 set ikev1 transform-set ESP-AES-256-SHA ESP-AES-192-SHA ESP-AES-128-SHA

crypto map outside_map 2 set ikev2 pre-shared-key *****

crypto map outside_map 4 match address outside_cryptomap_Site_D

crypto map outside_map 4 set peer 5.5.5.5

crypto map outside_map 4 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-192-SHA ESP-AES-256-SHA

crypto map outside_map 6 match address outside_cryptomap_US

crypto map outside_map 6 set peer 3.3.3.3

crypto map outside_map 6 set ikev1 transform-set ESP-AES-256-SHA ESP-AES-192-SHA ESP-AES-128-SHA ESP-3DES-SHA

crypto map outside_map interface outside



Regards



Alex

The sender of this email is an officer and / or employee of one of the JSP group of companies. The sender of this email is not authorised to legally bind any of the JSP companies by this email and none of such entities may be legally bound, except by a document signed on behalf of such entity by an authorised representative thereof.

This communication, including all attachments, is intended for the use of the individual to whom it is addressed and contains confidential information. If the reader is not the intended recipient, the reader is hereby notified.

ARPRO is a registered trade mark.