ISR4431 SEC Throughput Limit

[TONY SMITH]

Hi,

We have an ISR4431 which according to it's logs is hitting it's IPsec limit indicated by "Maximum Tx Bandwidth limit of 85000 Kbps reached for Crypto functionality with securityk9 technology package license."

Although this router does have some IPsec traffic from VPN tunnels, it does not appear that the amount of traffic on those tunnels should be enough to be tripping this limit as often as it seems.   So we're suspecting that the limit may not apply exactly as we thought.   Does anyone know if either of the following apply?

(1) Does the limit of 85meg apply to the total transmit traffic, so long as it includes some IPsec?  For example sending 10meg of IPsec + 80meg unencrypted.

(2) Does it include IPsec traffic passing through the router but not encrypted by it, for example traffic from a third-party VPN appliance?

Thanks, Tony S

[Joseph W. Doherty]

I would expect the limit only apply to all IPSec traffic sourced by the router.

When you say you're hitting the limit more often than expected, it might be due to microbursts. I.e. what happens at the millisecond level often is not seen in multi-second to multi-minute bandwidth utilization stats.

[TONY SMITH]

After monitoring for a while I think that might be correct.  Do you know what time interval the rate is measured over?

[Joseph W. Doherty]

For internal enforcement of bandwidth caps? No, sorry, I don't. It's likely in the millisecond range.

[jgrudier]

This thread is a little old but to follow up, the older versions of code sampled at a 10ms rate, the new versions of code around 16.x sample at 250ms rate.  This is for traffic going to the crypto engine for processing, so it would  only  be traffic that was either being encrypted or decrypted by the router,  and not encrypted traffic passing through the router

[carlosgfranco]

Hello Tony,

You're hitting a limitation that can be removed with the HSEC license. Going through its documentation you'll find the following:

(1) "With the HSEC-K9 license, the router can go over the curtailment limit of 225 tunnels maximum for IP Security (IPsec) and encrypted throughput of 85-Mbps unidirectional traffic"; the limit applies to encrypted throughput only.

(2) "The HSEC-K9 license removes the curtailment enforced by the U.S. government export restrictions on the encrypted tunnel count and encrypted throughput". Based on this, I'd say it applies to all encrypted traffic including that from 3rd Party Appliances.

Cheers,

Carlos

[TONY SMITH]

Thanks.  I was aware of the limit and of the HSEC licence option, however what I was hoping to do is to avoid buying that licence for an installation where IPsec throughput is in fact less than 85meg.  It shouldn't be needed.

I have had a response from Cisco TAC, saying firstly that the limit should apply only to traffic encrypted by the device, not to IPsec traffic passing through it but encrypted elsewhere.  Importantly they have also highlighted a number of bugs causing the limit to be incorrectly triggered by traffic below 85meg.   Finally I note that later IOS versions, needed to avoid these bugs, also raise the CERM limit to 250meg.   So that sounds like a win all round.

[carlosgfranco]

Hi Tony,

Thank you for sharing; I wasn't sure about the encrypted traffic from other devices although now that I've thought about it again, it makes sense. Glad to have a confirmed answer.

Cheers, Carlos

[Joseph W. Doherty]

"I have had a response from Cisco TAC, saying firstly that the limit should apply only to traffic encrypted by the device, not to IPsec traffic passing through it but encrypted elsewhere."

I wouldn't expect it to apply to transit traffic but you might double check it it only applies to outbound IPSec traffic sourced by the device. I could see ti also counting against traffic being received and decrypted by the device.

"Finally I note that later IOS versions, needed to avoid these bugs, also raise the CERM limit to 250meg."

I thought (?) I saw something recently that the limited was raised because the US government raised it, i.e. not due to avoiding bugs.

[TONY SMITH]

---

@Joseph W. Doherty wrote:
I wouldn't expect it to apply to transit traffic but you might double check it it only applies to outbound IPSec traffic sourced by the device. I could see ti also counting against traffic being received and decrypted by the device.

---

Everything that I've seen suggests a separate limit for received IPsec, again I understand this to apply only to traffic decrypted by the device.  For example ..

sh platform cerm-information
Crypto Export Restrictions Manager(CERM) Information:
 CERM functionality: ENABLED

 ----------------------------------------------------------------
 Resource                       Maximum Limit           Available
 ----------------------------------------------------------------
 Tx Bandwidth(in kbps)          85000                   85000
 Rx Bandwidth(in kbps)          85000                   85000
 Number of tunnels              225                     224
 Number of TLS sessions         1000                    1000

[Joseph W. Doherty]

Ah, that appears it counts ingress and egress separately and each against its own 85 Mbps.

[TONY SMITH]

With the software upgrade finally installed this is what it shows.  The new 250 meg limit if indeed there is one, is not mentioned.  On the other hand we haven't seen the limit message in the logs either since the upgrade.

#show platform software cerm
Crypto Export Restrictions Manager(CERM) Information:
 CERM functionality: ENABLED
 ----------------------------------------------------------------
 Resource                       Maximum Limit           Available
 ----------------------------------------------------------------
 Number of tunnels              225                     220
 Number of TLS sessions         1000                    1000
 Resource reservation information:
 D - Dynamic
 -----------------------------------------------------------------------
 Client         Tunnels    TLS Sessions
 -----------------------------------------------------------------------
 VOICE           0          0
 IPSEC           5          N/A
 SSLVPN          0          N/A
 Statistics information:
 Failed tunnels     : 0
 Failed sessions    : 0

[ChristopherCraddock66504]

Anyone have any idea why there were restrictions placed by the Govt in the first place? Seems strange that the Govt would regulate the # of tunnels you could have or the amount of traffic you could pump through a VPN. 

Thanks.

[Georg Pauwen]

Hello,

it comes down to National Security and Anti-Terrorism. It has it's origins in the WWII era; encryption is basically considered a 'weapon'.

Read the link below. You might have heard of the Enigma code used by German U-boats, the ultimate example of how encryption was used with deadly consequences.

https://en.wikipedia.org/wiki/Export_of_cryptography_from_the_United_States

[Joseph W. Doherty]

Don't know if they still do, but US Government (I recall?) classified certain encryption levels as munitions for export.  They didn't (I recall) care much about Mbps or number of tunnels, just how "good" (i.e. hard to break) the encryption was.

I think (?) the bandwidth limits was actually a way to ease up on the restrictions, i.e. for small volumes, they didn't care.

If what I remember is correct, you may still be wondering does it really make sense?  Well, I'm sure it does to the bureaucracy that defines all this.