Multiple SPI SAs on DMVPN Tunnel

[paul amaral]

Hi, Looking at the tunnel below I noticed I have 6 SPIs, 3 incoming and 3 outgoing. Although the tunnel is encrypting traffic both ways with no issues. I'm assuming I should only see 2 SPIs, one for each direction, so I'm curious why am seeing multiple SPIs. The other side is configured just like the router below. The interesting thing is if I clear the crypto session it will come up with just a pair of SPI but eventually it will go to multiple SPIs. This tunnel does constantly go to about 1 gig of encrypted traffic so am not sure is this is the intended behavior but it seems from the debugs below that the SPI are changing to frequently. This is on a Cisco ASR using cisco XE  Version 16.12.5. 

Can anyone help me figure out what is going on.

TIA, P 

#sh crypto isakmp policy

Global IKE policy
Protection suite of priority 2
        encryption algorithm:   AES - Advanced Encryption Standard (128 bit keys).
        hash algorithm:         Secure Hash Standard
        authentication method:  Pre-Shared Key
        Diffie-Hellman group:   #15 (3072 bit)
        lifetime:               3600 seconds, no volume limit

#sh crypto ipsec policy
No policy exists

#sh crypto ipsec profile
IPSEC profile ENS_ipsec_profile
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): N
        Mixed-mode : Disabled
        Transform sets={
                ipsec-transset-ENS:  { esp-aes esp-sha256-hmac  } ,
        }

IPSEC profile NB-Swansea-backup_profile
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): N
        Mixed-mode : Disabled
        Transform sets={
                ipsec-transset-ENS:  { esp-aes esp-sha256-hmac  } ,
        }

IPSEC profile NON-ENS_ipsec_profile
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): N
        Mixed-mode : Disabled
        Transform sets={
                ipsec-transset-NON-ENS:  { esp-aes esp-sha256-hmac  } ,
        }

IPSEC profile default
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): N
        Mixed-mode : Disabled
        Transform sets={
                default:  { esp-aes esp-sha-hmac  } ,
        }

IPSEC profile generic_ipsec_protection
        Security association lifetime: 4608000 kilobytes/3600 seconds
        Responder-Only (Y/N): N
        PFS (Y/N): N
        Mixed-mode : Disabled
        Transform sets={
                ipsec-transset-1:  { esp-aes esp-sha-hmac  } ,
        }

interface Tunnel0
 description xxx
 bandwidth 1000000
 ip address 10.2.2.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nbar protocol-discovery
 ip nhrp network-id 1
 ip nhrp redirect
 ip tcp adjust-mss 1360
 ip ospf network broadcast
 ip ospf priority 255
 ip ospf mtu-ignore
 ip ospf 1 area 0
 load-interval 30
 tunnel source GigabitEthernet0/0/2
 tunnel mode gre multipoint
 tunnel key 1
 tunnel path-mtu-discovery
 tunnel protection ipsec profile ENS_ipsec_profile shared


#sh crypto ipsec sa peer 172.17.3.2

interface: Tunnel0
    Crypto map tag: ENS_ipsec_profile-head-1-IPv4, local addr 172.17.3.1

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.17.3.1/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (172.17.3.2/255.255.255.255/47/0)
   current_peer 172.17.3.2 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 430462233, #pkts encrypt: 430462233, #pkts digest: 430462233
    #pkts decaps: 155241645, #pkts decrypt: 155241645, #pkts verify: 155241645
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 1560

     local crypto endpt.: 172.17.3.1, remote crypto endpt.: 172.17.3.2
     plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/2
     current outbound spi: 0x290E38E9(688797929)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0xCDD2533F(3453113151)
        transform: esp-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 19415, flow_id: HW:17415, sibling_flags FFFFFFFF80004008, crypto map: ENS_ipsec_profile-head-1-IPv4
        sa timing: remaining key lifetime (k/sec): (4608000/656)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)
      spi: 0xC14574C6(3242554566)
        transform: esp-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 19691, flow_id: HW:17691, sibling_flags FFFFFFFFC0004008, crypto map: ENS_ipsec_profile-head-1-IPv4
        sa timing: remaining key lifetime (k/sec): (4552005/3533)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)
      spi: 0x4F73B083(1332981891)
        transform: esp-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 19697, flow_id: HW:17697, sibling_flags FFFFFFFF80004008, crypto map: ENS_ipsec_profile-head-1-IPv4
        sa timing: remaining key lifetime (k/sec): (4553540/3568)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x50304C8(84083912)
        transform: esp-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 19416, flow_id: HW:17416, sibling_flags FFFFFFFF80004008, crypto map: ENS_ipsec_profile-head-1-IPv4
        sa timing: remaining key lifetime (k/sec): (4608000/656)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)
      spi: 0x50919C87(1351720071)
        transform: esp-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 19692, flow_id: HW:17692, sibling_flags FFFFFFFFC0004008, crypto map: ENS_ipsec_profile-head-1-IPv4
        sa timing: remaining key lifetime (k/sec): (884442/3533)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)
      spi: 0x290E38E9(688797929)
        transform: esp-aes esp-sha256-hmac ,
        in use settings ={Transport, }
        conn id: 19698, flow_id: HW:17698, sibling_flags FFFFFFFF80004008, crypto map: ENS_ipsec_profile-head-1-IPv4
        sa timing: remaining key lifetime (k/sec): (1302222/3568)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 1024
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:

I was able to get some isakmp debugs that show a constant SPI/SA install although I'm not sure why or if its related to DPD.

side A

Sep  7 23:11:13: ISAKMP: (79859):      SA life type in kilobytes
Sep  7 23:11:13: ISAKMP: (79859):      authenticator is HMAC-SHA256
Sep  7 23:11:13: ISAKMP: (79859):      key length is 128
Sep  7 23:11:13: ISAKMP: (79859):atts are acceptable.
Sep  7 23:11:13: ISAKMP: (79859):processing NONCE payload. message ID = 3666410542
Sep  7 23:11:13: ISAKMP: (79859):processing ID payload. message ID = 3666410542
Sep  7 23:11:13: ISAKMP: (79859):processing ID payload. message ID = 3666410542
Sep  7 23:11:13: ISAKMP: (79859):QM Responder gets spi
Sep  7 23:11:13: ISAKMP: (79859):Node 3666410542, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Sep  7 23:11:13: ISAKMP: (79859):Old State = IKE_QM_READY  New State = IKE_QM_SPI_STARVE
Sep  7 23:11:13: ISAKMP: (79859):Node 3666410542, Input = IKE_MESG_INTERNAL, IKE_GOT_SPI
Sep  7 23:11:13: ISAKMP: (79859):Old State = IKE_QM_SPI_STARVE  New State = IKE_QM_IPSEC_INSTALL_AWAIT
Sep  7 23:11:13: ISAKMP: (79859):Received IPSec Install callback... proceeding with the negotiation
Sep  7 23:11:13: ISAKMP: (79859):Successfully installed IPSEC SA (SPI:0xF7A925DB) on Tunnel0
Sep  7 23:11:13: ISAKMP-PAK: (79859):sending packet to 172.17.3.2 my_port 500 peer_port 500 (R) QM_IDLE
Sep  7 23:11:13: ISAKMP: (79859):Sending an IKE IPv4 Packet.
Sep  7 23:11:13: ISAKMP: (79859):Node 3666410542, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE
Sep  7 23:11:13: ISAKMP: (79859):Old State = IKE_QM_IPSEC_INSTALL_AWAIT  New State = IKE_QM_R_QM2
Sep  7 23:11:13: ISAKMP-PAK: (79859):received packet from 172.17.3.2 dport 500 sport 500 Global (R) QM_IDLE
Sep  7 23:11:13: ISAKMP: (79859):deleting node 3666410542 error FALSE reason "QM done (await)"
Sep  7 23:11:13: ISAKMP: (79859):Node 3666410542, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Sep  7 23:11:13: ISAKMP: (79859):Old State = IKE_QM_R_QM2  New State = IKE_QM_PHASE2_COMPLETE
Sep  7 23:11:27: ISAKMP: (79859):purging node 1383603721
Sep  7 23:11:37: ISAKMP-PAK: (79859):received packet from 172.17.3.2 dport 500 sport 500 Global (R) QM_IDLE
Sep  7 23:11:37: ISAKMP: (79859):set new node 3832189246 to QM_IDLE
Sep  7 23:11:37: ISAKMP: (79859):processing HASH payload. message ID = 3832189246
Sep  7 23:11:37: ISAKMP: (79859):processing DELETE payload. message ID = 3832189246
Sep  7 23:11:37: ISAKMP: (79859):peer does not do paranoid keepalives.
Sep  7 23:11:37: ISAKMP: (79859):Enqueued KEY_MGR_DELETE_SAS for IPSEC SA (SPI:0x83877863)
Sep  7 23:11:37: ISAKMP: (79859):deleting node 3832189246 error FALSE reason "Informational (in) state 1"
Sep  7 23:11:50: ISAKMP: (79859):set new node 0 to QM_IDLE
Sep  7 23:11:50: ISAKMP: (79859):SA has outstanding requests  (local 172.17.3.1 port 500, remote 172.17.3.2 port 500)
Sep  7 23:11:50: ISAKMP: (79859):sitting IDLE. Starting QM immediately (QM_IDLE      )
Sep  7 23:11:50: ISAKMP: (79859):beginning Quick Mode exchange, M-ID of 3247684825
Sep  7 23:11:50: ISAKMP: (79859):QM Initiator gets spi
Sep  7 23:11:50: ISAKMP-PAK: (79859):sending packet to 172.17.3.2 my_port 500 peer_port 500 (R) QM_IDLE
Sep  7 23:11:50: ISAKMP: (79859):Sending an IKE IPv4 Packet.
Sep  7 23:11:50: ISAKMP: (79859):Node 3247684825, Input = IKE_MESG_INTERNAL, IKE_INIT_QM
Sep  7 23:11:50: ISAKMP: (79859):Old State = IKE_QM_READY  New State = IKE_QM_I_QM1
Sep  7 23:11:50: ISAKMP-PAK: (79859):received packet from 172.17.3.2 dport 500 sport 500 Global (R) QM_IDLE
Sep  7 23:11:50: ISAKMP: (79859):processing HASH payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (79859):processing SA payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (79859):Checking IPSec proposal 1
Sep  7 23:11:50: ISAKMP: (79859):transform 1, ESP_AES
Sep  7 23:11:50: ISAKMP: (79859):   attributes in transform:
Sep  7 23:11:50: ISAKMP: (79859):      encaps is 2 (Transport)
Sep  7 23:11:50: ISAKMP: (79859):      SA life type in seconds
Sep  7 23:11:50: ISAKMP: (79859):      SA life duration (basic) of 3600
Sep  7 23:11:50: ISAKMP: (79859):      SA life type in kilobytes
Sep  7 23:11:50: ISAKMP: (79859):      authenticator is HMAC-SHA256
Sep  7 23:11:50: ISAKMP: (79859):      key length is 128
Sep  7 23:11:50: ISAKMP: (79859):atts are acceptable.
Sep  7 23:11:50: ISAKMP: (79859):processing NONCE payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (79859):processing ID payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (79859):processing ID payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (79859):Node 3247684825, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Sep  7 23:11:50: ISAKMP: (79859):Old State = IKE_QM_I_QM1  New State = IKE_QM_IPSEC_INSTALL_AWAIT
Sep  7 23:11:50: ISAKMP: (79859):Received IPSec Install callback... proceeding with the negotiation
Sep  7 23:11:50: ISAKMP: (79859):Successfully installed IPSEC SA (SPI:0x3C3C1CA4) on Tunnel0
Sep  7 23:11:50: ISAKMP-PAK: (79859):sending packet to 172.17.3.2 my_port 500 peer_port 500 (R) QM_IDLE
Sep  7 23:11:50: ISAKMP: (79859):Sending an IKE IPv4 Packet.
Sep  7 23:11:50: ISAKMP: (79859):deleting node 3247684825 error FALSE reason "No Error"
Sep  7 23:11:50: ISAKMP: (79859):Node 3247684825, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE
Sep  7 23:11:50: ISAKMP: (79859):Old State = IKE_QM_IPSEC_INSTALL_AWAIT  New State = IKE_QM_PHASE2_COMPLETE
Sep  7 23:11:50: ISAKMP: (79859):purging node 1388932311

Side B

Sep  7 23:11:50: ISAKMP-PAK: (75895):received packet from 172.17.3.1 dport 500 sport 500 Global (I) QM_IDLE
Sep  7 23:11:50: ISAKMP: (75895):set new node 3247684825 to QM_IDLE
Sep  7 23:11:50: ISAKMP: (75895):processing HASH payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (75895):processing SA payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (75895):Checking IPSec proposal 1
Sep  7 23:11:50: ISAKMP: (75895):transform 1, ESP_AES
Sep  7 23:11:50: ISAKMP: (75895):   attributes in transform:
Sep  7 23:11:50: ISAKMP: (75895):      encaps is 2 (Transport)
Sep  7 23:11:50: ISAKMP: (75895):      SA life type in seconds
Sep  7 23:11:50: ISAKMP: (75895):      SA life duration (basic) of 3600
Sep  7 23:11:50: ISAKMP: (75895):      SA life type in kilobytes
Sep  7 23:11:50: ISAKMP: (75895):      authenticator is HMAC-SHA256
Sep  7 23:11:50: ISAKMP: (75895):      key length is 128
Sep  7 23:11:50: ISAKMP: (75895):atts are acceptable.
Sep  7 23:11:50: ISAKMP: (75895):processing NONCE payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (75895):processing ID payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (75895):processing ID payload. message ID = 3247684825
Sep  7 23:11:50: ISAKMP: (75895):QM Responder gets spi
Sep  7 23:11:50: ISAKMP: (75895):Node 3247684825, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Sep  7 23:11:50: ISAKMP: (75895):Old State = IKE_QM_READY  New State = IKE_QM_SPI_STARVE
Sep  7 23:11:50: ISAKMP: (75895):Node 3247684825, Input = IKE_MESG_INTERNAL, IKE_GOT_SPI
Sep  7 23:11:50: ISAKMP: (75895):Old State = IKE_QM_SPI_STARVE  New State = IKE_QM_IPSEC_INSTALL_AWAIT
Sep  7 23:11:50: ISAKMP: (75895):Received IPSec Install callback... proceeding with the negotiation
Sep  7 23:11:50: ISAKMP: (75895):Successfully installed IPSEC SA (SPI:0x9EEF4211) on Tunnel0
Sep  7 23:11:50: ISAKMP-PAK: (75895):sending packet to 172.17.3.1 my_port 500 peer_port 500 (I) QM_IDLE
Sep  7 23:11:50: ISAKMP: (75895):Sending an IKE IPv4 Packet.
Sep  7 23:11:50: ISAKMP: (75895):Node 3247684825, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE
Sep  7 23:11:50: ISAKMP: (75895):Old State = IKE_QM_IPSEC_INSTALL_AWAIT  New State = IKE_QM_R_QM2
Sep  7 23:11:50: ISAKMP-PAK: (75895):received packet from 172.17.3.1 dport 500 sport 500 Global (I) QM_IDLE
Sep  7 23:11:50: ISAKMP: (75895):deleting node 3247684825 error FALSE reason "QM done (await)"
Sep  7 23:11:50: ISAKMP: (75895):Node 3247684825, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Sep  7 23:11:50: ISAKMP: (75895):Old State = IKE_QM_R_QM2  New State = IKE_QM_PHASE2_COMPLETE

[Cisco_Virtual_Engineer · Accepted solution]

Multiple Security Parameter Indexes (SPIs) in an IPsec VPN are used to handle different traffic flows between two endpoints. Each SPI signifies a unique security association, hence enabling secure communication to take place. The existence of multiple SPIs in an IPsec VPN offers several advantages:

1. Load Balancing: By utilizing multiple SPIs, load balancing of traffic across several security associations becomes possible. This approach helps in evenly spreading the traffic, thus preventing the possibility of a single SPI being overwhelmed.

2. Redundancy: In case one SPI fails or is unavailable, other SPIs can still process the traffic and maintain the VPN connection. This provides an element of redundancy and ensures that communication is not interrupted.

3. Scalability: The capability to handle a larger volume of traffic and support a greater number of VPN connections is made possible with multiple SPIs, thereby allowing for scalability as your network expands.

4. Security: Each SPI has its own distinct encryption and authentication key, which adds an extra layer of security. Even if one SPI is compromised, the others remain secure.

In summary, the presence of multiple SPIs in an IPsec VPN enhances performance, reliability, scalability, and security, this is achieved by distributing the traffic, providing redundancy, and supporting a larger number of secure connections.

[paul amaral]

hi, thanks for this great explanation. I suspected this was the case as multiple SPIs were only observed during high traffic, about 1 gig of encrypted traffic. I could not find any Cisco literature on the website that indicated what you stated above. 

Again, thanks for this explanation.

Paul