VPN error - %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed

[Brian Cartledge]

Getting this error on the data center 2581 (12.4(24)T) from a GRE/IPSEC tunnel, remote branch is 2811 running 12.4(25d)

%CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=

The tunnel has been up and working okay for months, nothing has changed on the config and the key is correct. Traffic is following but remote users are complaining of performance issues. A wireshark shows checksum errors and lots of packet resends. Remote ISP has checked the circuit and says its clean.

The data centre router has quite a few tunnels but only 1 causing this issue. From the head end router -

sh crypto ips sa | b x.x.x.x

   current_peer x.x.x.xport 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 15129, #pkts encrypt: 15129, #pkts digest: 15129
    #pkts decaps: 13346, #pkts decrypt: 13346, #pkts verify: 13346
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 1992

Can a VPN module go bad like this? I've tried disabling the branch onboard engine and using software but it doesn't help. Any ideas?

TIA

[wzhang]

Hi,

This is likely caused by packet corruption in the transit network, which is not all that unusual. To prove it, you could setup packet capture on the WAN side in front of each tunnel end point. You could also do this with the EPC (embedded packet capture) feature that was added to IOS in 12.4(20)T and later. Make sure you increase the packet buffer so that you can compare the entire packet. As soon as you see an MAC error logged on the router, stop the capture on both sides, and use the ip id, esp seq numbers to identifiy the packet in question from both captures. You can then compare them by printing the the packets to a file and doing a diff on them. If the packets are indeed identical, then you may want to open a TAC case to see if there's any known software/hardware issues that may cause this.

Thanks,

Wen