Help debugging funky log message

droeun141 · ‎03-30-2010

Noticed these as soon as I enabled the VSA module, so I'm not sure if the cause has always been present. We've been seeing TONS of them lately since pushing 80mb+ of replication traffic through the VPN. Cisco TAC says they can be normal and happen during a rekey, but I'm just curious if anyone else has experienced them before.

Mar 30 07:34:48.629 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0x17733,SPI=0x71CB70C5,
Mar 30 07:36:56.049 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0x52F78,SPI=0x71CB70C5,
Mar 30 07:50:29.985 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0xC2EC3,SPI=0xFCB1AF02,
Mar 30 08:01:50.291 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0xEAA3C,SPI=0xB0310E19,
Mar 30 08:04:18.641 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0x1B2F9,SPI=0x3E43A82B,
Mar 30 08:07:12.283 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0x7023D,SPI=0x3E43A82B,
Mar 30 08:10:25.075 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0xC5B76,SPI=0x3E43A82B,
Mar 30 08:14:32.190 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0x33A13,SPI=0x6120B1D6,
Mar 30 08:18:35.893 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0xA3AFA,SPI=0x6120B1D6,
Mar 30 08:20:35.925 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0xDA9F3,SPI=0x6120B1D6,
Mar 30 08:23:34.368 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0x308A6,SPI=0x4AF883FE,
Mar 30 08:24:06.030 EDT: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, MAC mismatch:srcadr=x.x.x.x,dstadr=x.x.x.x,size=104,sequence number=0x3D473,SPI=0x4AF883FE,

naveshar · ‎04-06-2010

Hi,

Here is a explanation of the error you are seeing:-

%VPN_HW-1-PACKET_ERROR: slot: [dec] Packet Encryption/Decryption error, [chars] 

An unexpected error occurred during the encryption or decryption of a packet.

Recommended Action: This message can occur occasionally during normal operation of the
system. It may occur during the transition to a new session key for a Security
Association. In such cases, no action is required. However, if it happens frequently, or
is associated with traffic disruption, we might be encountring hardware failure.

Could you please elaborate whether you are seeing these errors on the ASA/PIX or Router?

Also please get the output out command "show version". Also please get the output of command "show cry eng acc statistic ".

Also are you using DMVPN or GRE/IPsec or traditional Site to Site VPN when getting these errors?

Try adjusting the MSS size on the outside interface and see if we still see the same errors or not.

Just for your information this could also case due to faulty H/W module.

I think the best way to determine if the hardware module has gone bad would be
disabling the hardware encryption process.
It could be done by this command:
no crypto engine accelerator 

If this makes the error messages to go away,we would be sure the hardware module might
have gone bad and in that situation you should contact Cisco-TAC to get the RMA done for the faulty H/W module.

I hope this information helps.

Regards,

Naveen

Don't forget to rate this reply if it helps!

droeun141 · ‎04-06-2010

Hi Naveshar - thanks for the reply! If the error is referencing the module specifically - how will disabling it tell us if it's bad? Wouldn't the HW errors stop indefinitely since it switches to software? Please advise.

HRC-A.Intranet.VPN.Router#sh ver
Cisco IOS Software, 7200 Software (C7200P-ADVSECURITYK9-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 19-Oct-09 23:55 by prod_rel_team

ROM: System Bootstrap, Version 12.4(12.2r)T, RELEASE SOFTWARE (fc1)
BOOTLDR: Cisco IOS Software, 7200 Software (C7200P-KBOOT-M), Version 12.4(24)T2, RELEASE SOFTWARE (fc2)

HRC-A.Intranet.VPN.Router uptime is 10 weeks, 2 days, 3 hours, 47 minutes
System returned to ROM by reload at 03:32:19 EST Sun Jan 24 2010
System restarted at 03:33:48 EST Sun Jan 24 2010
System image file is "disk2:c7200p-advsecurityk9-mz.124-24.T2.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 7206VXR (NPE-G2) processor (revision A) with 917504K/65536K bytes of memory.
Processor board ID 36161919
MPC7448 CPU at 1666Mhz, Implementation 0, Rev 2.2
6 slot VXR midplane, Version 2.11

Last reset from power-on

PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.
Current configuration on bus mb1 has a total of 0 bandwidth points.
This configuration is within the PCI bus capacity and is supported.

PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.
Current configuration on bus mb2 has a total of 0 bandwidth points.
This configuration is within the PCI bus capacity and is supported.

Please refer to the following document "Cisco 7200 Series Port Adaptor
Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com>
for c7200 bandwidth points oversubscription and usage guidelines.

1 FastEthernet interface
3 Gigabit Ethernet interfaces
1 Virtual Private Network (VPN) Module
2045K bytes of NVRAM.

250880K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).
65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

HRC-A.Intranet.VPN.Router#show cry eng acc statistic ?
<0-7> Card slot number

HRC-A.Intranet.VPN.Router#show cry eng acc statistic 0

Device: VSA
Location: Service Adapter: 0
VSA Traffic Statistics

Inbound rate: 277pps 523kb/s Outbound rate: 356pps 2281kb/s

    TRAFFIC                     Transmitted                  Received
-------------------------------------------------------------------------------
Message Count:                   1523209                   1523209
Message Byte Count:            277991777                 558957196
Message Overflow:                      0
Outbound Count:                9207500097                9217940263
Outbound Byte Count:       11147272579789            11868845107414
Outbound Overflow:                 554483
Inbound Count:                4573683386                4573683084
Inbound Byte Count:         784653586520              661424863427
Inbound Overflow:                      0

Reassembled Pkt:                     2901
Fragments Dropped:                      0
     IPPE:                                0
     EPPE:                                0
     FIFO:                                0
     RAE:                                 0

Inbound Traffic:
-------------------------------------------------------------------------------
Decrypted Pkt:                 4573135115
Passthrough Pkt:                   152976
IKE Pkt:                           117163

SPI Error: 269060
Policy Violation: 0

Outbound Traffic:             Route cache                 Processor
-------------------------------------------------------------------------------
Encrypted Pkt:                 9217358292                     80766
Passthrough Pkt:                   206397                    189733
Policy Violation:                  105064

Queue Depth:
------------------------------------------------------------------------------
TXRing Current Queue Depth:
    High Priority   :                     0.0 %
    Medium Priority :                     0.0 %
    Low Priority    :                     0.0 %

VSA RX Exception statistics:
   Invalid SA              :          0   Enc Dec mismatch        :          0
   Next Header mismatch    :          0   Pad mismatch            :          0
   MAC mismatch            :       5749   Anti replay failed      :        275
   Enc Seq num overflow    :          0   Dec IPver mismatch      :          0
   Enc IPver mismatch      :          0   TTL Decr                :          0
   Selector checks         :          0   UDP mismatch            :          0
   IP Parse error          :          0   Fragmentation Error     :          0
   IB Selector check       :          0   TimeBased Replay Err    :          0
   Misc. Exceptions        :          0

naveshar · ‎04-06-2010

Hi,

Could you please confirm if you are using VAM or VAM2 crypto card on the router. Get the output of "show diag" command on the router.Looking at the output of "show cry eng acc statistic 0" it appears that H/W module is seeing alot of VPN traffic and it's buffers is getting overflow. Also we are seeing alot of "MAC mismatch : 5749 Anti replay failed : 275" errors which suggests the h-MAC verification is failing time to time. Could you check if these errors go away itself if we disable the H/W crypto engine and when all the VPN tunnels switch back to software encryption engine instead?

Regards,

Naveen.

droeun141 · ‎04-06-2010

The errors only started after we enabled the VSA. Prior to that it was doing it in pure software with no errors.

HRC-A.Intranet.VPN.Router#sh diag
Slot 0:
        VSA IPsec Card Port adapter
        Port adapter is analyzed
        Port adapter insertion time 10w2d ago
        EEPROM contents at hardware discovery:
        Hardware Revision        : 1.0
        Part Number              : 73-10220-05
        Board Revision           : B0
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Deviation Number         : 0
        Product (FRU) Number     : C7200-VSA
        Version Identifier       : V01
        Top Assy. Part Number    : 68-2578-05
        CLEI Code                : CNUCAFNAAA
        EEPROM format version 4
        EEPROM contents (hex):
          0x00: 04 FF C1 8B 4A 41 42 31 31 35 31 30 31 52 56 40
          0x10: 05 0D 41 01 00 82 49 27 EC 05 42 42 30 03 00 81
          0x20: 00 00 00 00 04 00 88 00 00 00 00 CB 94 43 37 32
          0x30: 30 30 2D 56 53 41 20 20 20 20 20 20 20 20 20 20
          0x40: 20 89 56 30 31 20 D9 03 C1 40 CB 87 44 0A 12 05
          0x50: C6 8A 43 4E 55 43 41 46 4E 41 41 41 FF FF FF FF
          0x60: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
          0x70: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF