cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3160
Views
5
Helpful
18
Replies

VPN Spoke-Hub-Spoke Troubleshooting

aw3000
Level 1
Level 1

Hi all,

 

I am having some problems with a spoke-hub-spoke setup and I'm really stumped.  Hopefully some knowledgeable person can help me out...

I have three sites: a hub and spokes A and B.  Site-to-site VPNs are configured between spoke A and the hub, and spoke B and the hub.  The hub can communicate with each spoke, but they can't communicate with each other.

Things I have already done:

Spoke A ASA: added the spoke B subnets to the interesting traffic and no-NAT

Spoke B ASA: added the spoke A subnets to the interesting traffic and no-NAT

Hub ASA: configured same-security-traffic inter-interface; configured no-NAT from the spoke A subnets to the spoke B subnets on the outside interface

The only routing in place on the hub ASA is 0.0.0.0 to the outside interface.

Using the command show crypto sa peer <peer ip> on the hub ASA, I can see that traffic for spoke B is arriving from spoke A and being de-encapsulated, but not being encapsulated and sent to spoke B.  Same for traffic from spoke B to spoke A.

I will post santised configs if required, but is there anything I could have missed?

 

Many thanks

 

Alex

 

Spoke-Hub-Spoke.jpg

18 Replies 18

Rahul Govindan
VIP Alumni
VIP Alumni

Spoke A ASA: added the spoke B subnets to the interesting traffic and no-NAT

Spoke B ASA: added the spoke A subnets to the interesting traffic and no-NAT

Hub ASA: configured same-security-traffic inter-interface; configured no-NAT from the spoke A subnets to the spoke B subnets on the outside interface

 

Did you add Spoke A subnet as local network interesting traffic for Spoke B tunnel and vice versa on the hub? The ACL's should be mirror images on both sides. 

Hi Rahul, yes I did.  I forgot to incude that, sorry!

Can you run a packet trace on each node for a packet from A-Subnet to B-Subnet and another one from B-Subnet to A-Subnet? This will help to see what is happening to the packet.

Hi Mohammed,

 

I have tried packet tracer before but it doesn't always give accurate results.  For example: the screenshot below is on the Spoke A ASA, and it shows traffic from the Hub ASA local subnet being dropped, which isn't what really happens.

Is it correct to choose the Outside interface when packet tracing inbound VPN traffic?

 

Many thanks

 

Alex

 

Capture.PNG

You need to have active sa before running packet trace to match vpn encrypt action. Also, i asked for a trace from sub-A to sub-B and vice versa which matches your subnet

Ok, point taken!  Here are the results of Spoke A to Spoke B:

 

Spoke A ASA: Packet allowed

Capture.PNG

Hub ASA: Packet dropped

Capture.PNG

 

And here are the results of Spoke B to Spoke A:

Spoke B ASA: Packet allowed

Capture.PNG

 

Hub ASA: Packet dropped

Capture.PNG

 

Those results echo what I have seen in the VPN tunnel stats on the hub.  Traffic from Spoke A to Spoke B and vice-versa has 0 encapsulated packets.

 

Regards


Alex

 

On the hub, can you share the full trace to see why packet is dropped

Sure, here you are:

 

Result of the command: "packet-tracer input outside icmp 10.25.1.180 8 0 10.12.12.180 detailed"

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 via 217.33.171.225, outside

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,outside) source static EU_Networks EU_Networks destination static NA_Networks NA_Networks no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface outside
Untranslate 10.12.12.180/0 to 10.12.12.180/0

Phase: 3
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fff35bbcdd0, priority=11, domain=permit, deny=true
hits=48241194, user_data=0x5, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=outside, output_ifc=any

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Try "packet-tracer input outside icmp 10.25.1.180 8 0 10.12.12.180 detailed decrypted"

If you don't use "decrypted" , packet-tracer assumes that it is non-vpn traffic  sourced from the outside interface. This usually hits the inbound ACL on the outside interface. VPN traffic by default bypasses this ACL, so you have to use the decrypted keyword to simulate that. 

 

Also, share a sanitized version of the relevant configs (crypto map, acl, NAT) from all 3 sites if possible. 

Unfortunately 'decrypted' doesn't work - from the Cisco command reference, it was only introduced in IOS 9.9.(1) and we are currently running 9.2(2)4.  I will arrange an upgrade, but in the meantime I will work on the sanitised configs.  I will update the thread as soon as I can!  Many thanks for your help so far.


Alex

Hi there, please find the sanitised configs attached.  Sorry for the delay - I was on vacation.

 

 

Many thanks!

 

Alex

Hi Rahul,

 

I posted the sanitised configs above.  I'd be hugely grateful if you could take a look over them as I am truly baffled as to why this is not working.

Unfortunately I haven't been able to update to IOS 9.9 yet as this is a production environment.

 

Regards

 

Alex

Config looks correct to me. 

 

Can you get the output of "show crypto ipsec sa" from the hub? Also, try running the following debugs on the hub to see if it is attempting to establish a phase 2 tunnel.

 

debug crypto condition peer x.x.x.x

debug crypto ikev1 127

debug crypto ipsec 127

Hi Rahul,



The Hub-Site A and Hub-Site B tunnels are up. When I run show crypto ipsec sa I see entries for each subnet pair. For example: 10.25.0.0 (Site A) to 10.10.0.0 (Site B), 10.10.0.0 (Site B) to 10.25.0.0 (Site A) and 10.50.0.0 (Hub) to 10.25.0.0 (Site A):



Crypto map tag: outside_map, seq num: 6, local addr: 1.1.1.1



access-list outside_cryptomap_SiteB extended permit ip 10.25.0.0 255.255.0.0 10.10.0.0 255.255.0.0

local ident (addr/mask/prot/port): (10.25.0.0/255.255.0.0/0/0)

remote ident (addr/mask/prot/port): (10.10.0.0/255.255.0.0/0/0)

current_peer: 3.3.3.3



#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 232, #pkts decrypt: 232, #pkts verify: 232

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0

#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0

#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

#TFC rcvd: 0, #TFC sent: 0

#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0

#send errors: 0, #recv errors: 0



local crypto endpt.: 1.1.1.1/0, remote crypto endpt.: 3.3.3.3/0

path mtu 1500, ipsec overhead 74(44), media mtu 1500

PMTU time remaining (sec): 0, DF policy: copy-df

ICMP error validation: disabled, TFC packets: disabled

current outbound spi: B0DA1E28

current inbound spi : 88B61C10



inbound esp sas:

spi: 0x88B61C10 (2293636112)

transform: esp-aes-256 esp-sha-hmac no compression

in use settings ={L2L, Tunnel, IKEv1, }

slot: 0, conn_id: 76800000, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (4373955/17562)

IV size: 16 bytes

replay detection support: Y

Anti replay bitmap:

0xFFFFFFFF 0xFFFFFFFF

outbound esp sas:

spi: 0xB0DA1E28 (2967084584)

transform: esp-aes-256 esp-sha-hmac no compression

in use settings ={L2L, Tunnel, IKEv1, }

slot: 0, conn_id: 76800000, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (4374000/17562)

IV size: 16 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001



Crypto map tag: outside_map, seq num: 2, local addr: 1.1.1.1



access-list outside_cryptomap_SiteA extended permit ip 10.10.0.0 255.255.0.0 10.25.0.0 255.255.0.0

local ident (addr/mask/prot/port): (10.10.0.0/255.255.0.0/0/0)

remote ident (addr/mask/prot/port): (10.25.0.0/255.255.0.0/0/0)

current_peer: 2.2.2.2



#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 623, #pkts decrypt: 623, #pkts verify: 623

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0

#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0

#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

#TFC rcvd: 0, #TFC sent: 0

#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0

#send errors: 0, #recv errors: 0



local crypto endpt.: 1.1.1.1/0, remote crypto endpt.: 2.2.2.2/0

path mtu 1500, ipsec overhead 74(44), media mtu 1500

PMTU time remaining (sec): 0, DF policy: copy-df

ICMP error validation: disabled, TFC packets: disabled

current outbound spi: 41877D0F

current inbound spi : 0290153C



inbound esp sas:

spi: 0x0290153C (42997052)

transform: esp-aes-256 esp-sha-hmac no compression

in use settings ={L2L, Tunnel, IKEv1, }

slot: 0, conn_id: 76795904, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (4373853/12428)

IV size: 16 bytes

replay detection support: Y

Anti replay bitmap:

0xFFFFFFFF 0xFFFFFFFF

outbound esp sas:

spi: 0x41877D0F (1099398415)

transform: esp-aes-256 esp-sha-hmac no compression

in use settings ={L2L, Tunnel, IKEv1, }

slot: 0, conn_id: 76795904, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (4374000/12428)

IV size: 16 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001



Crypto map tag: outside_map, seq num: 2, local addr: 1.1.1.1



access-list outside_cryptomap_SiteA extended permit ip 10.50.0.0 255.255.0.0 10.25.0.0 255.255.0.0

local ident (addr/mask/prot/port): (10.50.0.0/255.255.0.0/0/0)

remote ident (addr/mask/prot/port): (10.25.0.0/255.255.0.0/0/0)

current_peer: 2.2.2.2



#pkts encaps: 9153909, #pkts encrypt: 9153909, #pkts digest: 9153909

#pkts decaps: 12535748, #pkts decrypt: 12535715, #pkts verify: 12535715

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 9153909, #pkts comp failed: 0, #pkts decomp failed: 0

#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0

#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

#TFC rcvd: 0, #TFC sent: 0

#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0

#send errors: 0, #recv errors: 33



local crypto endpt.: 1.1.1.1/0, remote crypto endpt.: 2.2.2.2/0

path mtu 1500, ipsec overhead 74(44), media mtu 1500

PMTU time remaining (sec): 0, DF policy: copy-df

ICMP error validation: disabled, TFC packets: disabled

current outbound spi: 2ABDE286

current inbound spi : 033CB0E8



inbound esp sas:

spi: 0x033CB0E8 (54309096)

transform: esp-aes-256 esp-sha-hmac no compression

in use settings ={L2L, Tunnel, IKEv1, }

slot: 0, conn_id: 76795904, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (4373159/28326)

IV size: 16 bytes

replay detection support: Y

Anti replay bitmap:

0xFFFFFFFF 0xFFFFFFFF

outbound esp sas:

spi: 0x2ABDE286 (717087366)

transform: esp-aes-256 esp-sha-hmac no compression

in use settings ={L2L, Tunnel, IKEv1, }

slot: 0, conn_id: 76795904, crypto-map: outside_map

sa timing: remaining key lifetime (kB/sec): (4372159/28326)

IV size: 16 bytes

replay detection support: Y

Anti replay bitmap:

0x00000000 0x00000001



>From the Hub, I can ping hosts in Site A and Site B - howewer, hosts in Site A can't ping hosts in Site B and vice-versa. If you look at the stats in the first two Cryptomaps above, you can see that no packets are being encapsulated: #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

So for some reason, packets sent from Site A to Site B (for example) are reaching the Hub and being decrypted, but not being encrypted and sent to Site B.



Regards



Alex


Review Cisco Networking products for a $25 gift card