cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
3274
Views
0
Helpful
8
Replies
Highlighted
Not applicable

CISCO 2901 ISRG2 CISCO IPsec throughput getting only 4Mbps

Hi Everyone,

Context:

1) CISCO 2901 ISRG2 configured in DMVPN, traffic encrypted with CISCO IPSec authenticated through Cert Based Authentication(authentication rsa-sig) through mGRE Tunnel.

- Error received: %CERM-4-TX_BW_LIMIT: Maximum Tx Bandwidth limit of 85000 Kbps reached for Crypto functionality with securityk9 technology package license.

- Actual encrypted throughput received in LAB environment with file transfer(>250Mb file) end to end is only about 4Mbps.

- CPU Utilization is only about 30%

CA server is the Hub Router(CISCO IOS CA Server)

ISAKMP Phase1

crypto isakmp policy 1

encry aes

hash sha256

authentication rsa-sig

ISAKMP Phase2

crypto ipsec transform-set TRANSFORM_SET esp-aes esah-sha256-hmac

mode transport

2) This aside i would also would like to know how to check if my router actually has an ISM(Internal Service Module) for VPN, via CLI, and what the results would be displayed as.

Regards,

Faizal

8 REPLIES 8
Highlighted
Cisco Employee

Hi Faizal,

CERM bandwidth limit reached indicates that the maximum TX bandwidth limit for the crypto functionality has been reached. The indicated TX bandwidth is the maximum that is allowed with your securityk9 license.

 

Due to strong crypto export restrictions enforced by the United States government, a securityk9 license only allows payload encryption up to rates close to 90 Megabits per second (Mbps) and limits the number of encrypted tunnels/Transport Layer Security (TLS) sessions to the device. 85Mbps is enforced on Cisco devices.

 

What often happens is that the device sends a huge burst of traffic (usually in a fraction of a second) over the physical interface, but then gets quickly rate-limited down to the actual supported speed. 85 Mbps TX is assigned to the ENTIRE box - it is not per interface.

Even if the burst is for a few milliseconds, it is enough to trigger the curtailed crypto bandwidth limit. And in these situations, the traffic that exceeds 85Mbps is dropped.

 

The router takes the incoming traffic, decides if it should be encrypted outbound, and then does the CERM calculations.

 

Let's say the inside network is attached to a GigabitEthernet port. If 200 Mbps of data comes in and the router decides it should all be encrypted, the router will drop said traffic before it even begins transmitting on the outbound interface. CERM occurs before outbound encryption by design, so that once it reaches the 85,000 Kb limit packets can no longer encrypt.

 

What we can do is, we can throttle the traffic down either at the application point or at a device before the router.

 

But, you can use HSEC-K9 license in order to mitigate this issue permanently.

 

With the HSEC-K9 license, the router can go over the curtailment limit of 225 tunnels maximum for IP Security (IPsec) and encrypted throughput

of 85-Mbps unidirectional traffic in or out of the ISR G2 router, with a bidirectional total of 170 Mbps.

 

For HSEC Licensing Please refer to document given below: -

 

http://www.cisco.com/c/dam/en/us/products/collateral/routers/3900-series-integrated-services-routers-isr/qa_c67_606268.pdf

 

 

Do you see any drops or latency when you get this error on the router ?.

 

CERM limit can be easily hit with micro bursts, the counters may not be able to catch these bursts given the sampling granularity. CERM logically divides traffic into fixed time intervals (200 msec on the ISR, IIRC).  It has a fixed budget for each time interval (which is the amount of traffic we're allowed to transmit over the wire that the maximum CERM rate within that time interval); if a burst of traffic causes us to use up the 'budget' for that time interval, we essentially block traffic until the time interval is over (and we get more budget).

 

Enhancement request CSCuz49319 has been filed to get more insight into how exactly micro-bursts are dealt when it comes to CERM limit.

 

     https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz49319/?reffering_site=dumpcr

 

 

To troubleshoot issue where Bandwidth CERM Limit is Reached, we can do the below steps:

 

Mirror the traffic on the connected switch.

 

Use Wireshark in order to analyse the captured trace by going down to two to 10 msec time period granularity.

 

Traffic with micro bursts greater than 85Mbps is an expected behaviour.

 

In a nutshell, if you are experiencing any packet-loss when these logs are being printed, you will need to troubleshoot the packet loss issue. If this never runs above 85Mbps crypto traffic, you may simply ignore the CERM logs.

Regards,

Aditya 

Please rate helpful posts and mark correct answers

Highlighted

Hi Aditya,

Thank you for your reply, but i am not understanding this.

For example, once tunnel protection have been turned off the routers(CISCO 2901 ISRG2) are able to reach up to speeds of about 20Mbps, during file transfer between clients at both ends, which is relatively more than what CISCO IPSec( with tunnel protection) can provide right now(3-4Mbps) for me. These are wire speed tests setup through Ethernet in a lab environment and not even deployed out to WAN yet, and also i am not seeing the traffic going anywhere between 40Mbps to 80Mbps.

Hence i would prefer to have a solution that does not require any additional paid license if possible as i do not see the license being an issue other than the possible microburst that youre mentioning? And if i may ask, how can i then validate that the latency of 3Mbps-4Mbps, is actually due to the microburst and license issue. Is there are trial license that would allow me to see a contrast of differences once this temporary license is installed so that i can state specifically that it is indeed a license issue(e.g perhaps after license is installed i could actually see improvement on encrypted traffic to go beyond 30Mbps?).

2) This aside, i've run "sh cry eli", and im able to see:

Hardware Encryption: Active

Number of hardware crypto engine = 1

CryptoEngine Onboard VPN details: state = Active

Capability: IPPCP, DES, 3DES, AES, GCM, GMAC, IPv6, GDOI, FAILCLOSE, HA

IPSec- Session: 0 active, 2800 max, 0 failed

Does this mean that i have ISM for installed and that this hardware is already active and running for CISCO IPSec VPN?

Appreciate very much for the assistance.

Regards,

Faizal

Highlighted

Hi Faizal,

Since you are getting relatively low speeds while file transfers through VPN tunnels you can try reducing the tunnel mtu under the tunnel interface and check.

Regarding your second query can you share the output of sh cry engine brief?

More info on this link:

https://supportforums.cisco.com/document/142291/verifying-cisco-vpn-internal-service-module-vpn-ism-installation-and-operations

Regards,

Aditya

Highlighted

Hi Aditya,

Thank you for your reply again, really need your help on this.

1) On the part of the sh invent i do not see "ISM-VPN-29" but instead "C2901 Mother board 2GE, integrated VPN and 4W on Slot 0" does this mean my router do not have an Internal Service Module for VPN?

2) Tunnel are already configured with ip mtu 1400 and tcp adjust-mss 1360

The following are the outputs.

Hub-R1#sh inventory raw
NAME: "CISCO2901/K9", DESCR: "CISCO2901/K9 chassis, Hw Serial#: FGL210810Z9, Hw Revision: 1.0"
PID: CISCO2901/K9 , VID: V06 , SN: FGL210810Z9

NAME: "C2901 Chassis Slot 0", DESCR: "C2901 Chassis Slot"
PID: , VID: , SN:

NAME: "C2901 Mother board 2GE, integrated VPN and 4W on Slot 0", DESCR: "C2901 Mother board 2GE, integrated VPN and 4W"
PID: , VID: V06 , SN: FOC21030EAW

Hub-R1#sh crypto engine br
crypto engine name: Virtual Private Network (VPN) Module
crypto engine type: hardware
State: Enabled
Location: onboard 0
Product Name: Onboard-VPN
HW Version: 1.0
Compression: Yes
DES: Yes
3 DES: Yes
AES CBC: Yes (128,192,256)
AES CNTR: No
Maximum buffer length: 0000
Maximum DH index: 0000
Maximum SA index: 0000
Maximum Flow index: 2800
Maximum RSA key size: 0000


crypto engine name: Cisco VPN Software Implementation
crypto engine type: software
serial number: 49E0B6E5
crypto engine state: installed
crypto engine in slot: N/A


Hub-R1#sh running-config interface tu10
interface Tunnel10
description traffic
ip address 1.1.1.1 255.255.255.0
no ip redirects
no ip unreachables
ip mtu 1400
ip nhrp authentication auth
ip nhrp map multicast dynamic
ip nhrp network-id 111
ip nhrp holdtime 60
ip nhrp registration no-unique
ip nhrp registration timeout 30
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 222
tunnel path-mtu-discovery
tunnel protection ipsec profile ipsec

Highlighted

Hi,

As per the output, I do not see the ISM installed.

Also, do you have the correct set of licenses installed on the device?

You can configure ‘ip mtu 1350’ and check if that helps to improve the speeds.

Regards,

Aditya

Highlighted

Hi Aditya,

Thank you for the prompt response.

1) Noted on the ISM VPN not being present in the routers

2) I've adjusted ip mtu to 1350 with no improvements

3) Can i then validate that the latency of between 3Mbps to 4Mbps encrypted traffic is correct  due to the fact that ISM VPN is not present in the routers?

4) The following is the sec-k9 license that i have on my routers

Hub-R1#sh license
Index 1 Feature: ipbasek9
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium
Index 2 Feature: securityk9
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium

Highlighted
Cisco Employee

Hi Faizal,

For your second query you can check :

sh cry eli and sh inventory

Regards,

Aditya

Highlighted
Beginner

OP,

 

This was a couple of years ago, did you ever get this figured out?

Content for Community-Ad