Cisco 1921 ISR IPsec failing with invalid SPI

[T.K. Welling]

Hello,

We have a customer who's experiencing problems with their ipsec tunnels.

Details:

1 Mainsite in our datacenter.

4 Remote sites, 1 is our office for testing/troubleshooting purpose.

All remote sites have an ipsec tunnel to our Data center.

1 tunnel between Site C (customer) and Site D (our office) for our troubleshooting purpose.

Al 5 routers are 1921 ISR routers.

Firmware: Version 15.2(4)M5

The 3 remote sites from our customer are (kpn business ) fiber pppoe internet connections. Our office is a Cable internet connection. All with static ip.

The problem is that one of the customer 3 tunnel's is randomly having a decapsulation error log:

: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=3.3.3.3, prot=50, spi=0xE3729320(3815936800), srcaddr=100.100.100.100, input interface=Dialer0

It looks like the SA negotiations are getting out of sync somehow, although we can’t find out why this is happening.

We have:

    Replaced a remote site router
    Replaced the Ma

[T.K. Welling]

Hello an update,

We have reviewed de debug logs and found out the following:

It looks like sometimes SiteC's SA is about to expire and  try's to negotiate with the MainSite. But the Main site's SA is about to expire half a minute later. While CiteC is trying to negotiate with the MainSite, the MainSite isn’t responding and logging anything at all. At this moment the “decaps: rec'd IPSEC packet has invalid spi” accrues. About half a minute later (when the MainSite’s SA expires) The negotiation happens all over again and succeeds.

30 seconds is about the same time as the tunnel go’s down.

Although it looks like we have found the source of the problem, we can’t figure out why this is happening and how to solve this problem.

SiteA and B are also experiencing the same problem, SiteD which has a cable connection is fine. Also the ipsec tunnel (for testing purpose) between SiteC and SiteD work’s fine.

See the attachments for the debug log’s.