Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1545
Views
0
Helpful
2
Replies

2911 + ISM-VPN-29 Accelerator tunnel throughput abnormal

wayfaring
Level 1
Level 1

‎08-20-2016 06:48 AM - edited ‎03-05-2019 04:32 AM

To sum up: 2911 dmvpn tunnel egress throughput increased slightly and there is less cpu usage after activating the ISM-VPN card, but dmvpn tunnel ingress throughput actually decreased and there is more cpu usage.  The expectation was that both ingress and egress throughput would have increased substantially after activating the card based on Cisco documentation, any suggestions are appreciated.



Expectation for tunneled ipsec traffic without ISM-VPN card installed -
• The SEC-K9 license limits encrypted throughput to less than or equal to 85-Mbps unidirectional traffic in or out of the ISR G2 router, with a bidirectional total of 170 Mbps. This requirement applies for the Cisco 1900, 2900, and 3900 ISR G2 platforms.

Expectation for tunneled ipsec traffic with ISM-VPN card installed, low IMIX number being the real world number:
• The Cisco 2900 Series Module (ISM-VPN-29) can provide hardware-based IPSec encryption services of 145 and 550 Mbps in the Cisco 2901, 150 and 600 Mbps in the Cisco 2911, 220 and 700 Mbps in the Cisco 2921, and 385 and 900 Mbps in the Cisco 2951 (IPSec IMIX and 1400-byte packets).


spoke is the 2911 + ISM-VPN card being tested
hub is an ASR1002 nat'd behind an ASA5520, no changes are being made at the hub during these tests.
dmvpn throughput with no Crypto engine slot 0  or before adding the card = ~50Mb up from spoke to hub, ~25-30Mb down from hub to spoke
dmvpn throughput with Crypto engine slot 0 after adding the card = ~65-70Mb up from spoke to hub, ~15-20Mb down from hub to spoke
numbers came up similar in repeat tests with iperf, smb, ftp transfers over an extended period


spoke isp bandwidth 200Mb/200Mb, internet non-tunnel speed test from lan test machine = 125Mb down / 125Mb up (2911 performance limitation)
hub isp bandwidth 500Mb/500Mb, internet non-tunnel speed test from lan test machine = 450Mb down / 375Mb up
unencrypted peering throughput test between these locations is still pending.



HUB -

NAME: "Chassis", DESCR: "Cisco ASR1002 Chassis" PID: ASR1002
NAME: "module F0", DESCR: "Cisco ASR1000 Embedded Services Processor, 5Gbps" PID: ASR1000-ESP5      , VID: V03,
NAME: "module 0", DESCR: "Cisco ASR1002 SPA Interface Processor 10" PID: ASR1002-SIP10
NAME: "subslot 0/0 transceiver 0", DESCR: "GE T" PID: SP7041-E          , VID: E  
NAME: "module R0", DESCR: "Cisco ASR1002 Route Processor 1" PID: ASR1002-RP1       , VID: V05,

sh ver hub - asr1000rp1-adventerprisek9.03.13.05.S.154-3.S5-ext.bin

hub -

crypto ipsec security-association replay window-size 1024
! Warning: window size of 512 actually used
!
crypto ipsec transform-set DMVPNtrans esp-des esp-md5-hmac
 mode transport
crypto ipsec df-bit clear
!

interface Tunnel0
 ip address 192.168.0.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 no ip split-horizon eigrp 100
 no ip next-hop-self eigrp ***
 ip nhrp authentication ***
 ip nhrp map multicast dynamic
 ip nhrp network-id ***
 ip nhrp holdtime 300
 ip nhrp max-send 200 every 10
 ip nhrp redirect
 ip tcp adjust-mss 1360
 load-interval 30
 delay 10000000
 qos pre-classify
 keepalive 10 3
 cdp enable
 tunnel source Loopback0
 tunnel mode gre multipoint
 tunnel key ****
 tunnel bandwidth transmit 5000000
 tunnel bandwidth receive 5000000
 tunnel protection ipsec profile ***
!

interface Loopback0
 description hub receives ASA public internet nat here
 ip address 192.168.248.1 255.255.255.240
 no ip redirects
 no ip unreachables
 ip mtu 1400
!

interface GigabitEthernet0/0/0
 description hub lan
 ip address 192.168.0.210 255.255.252.0
 no ip redirects
 no ip unreachables
 ip access-group external-security in
 negotiation auto
!




everything below  is from the spoke -


sh inv -
NAME: "CISCO2911/K9", DESCR: "CISCO2911/K9 chassis, Hw Serial#: , Hw Revision: 1.0"
NAME: "Internal Services Module - Crypto Engine on Slot 0", DESCR: "Internal Services Module - Crypto Engine" PID: ISM-VPN-29        , VID: V02 , SN:


sh ver   - running c2900-universalk9-mz.SPA.152-4.M1.bin


sh lic -
Index 9 Feature: hseck9
        Period left: Life time
        License Type: Permanent
        License State: Active, In Use
        License Count: Non-Counted
        License Priority: Medium
Index 2 Feature: securityk9
        Period left: Life time
        License Type: Permanent
        License State: Active, In Use
        License Count: Non-Counted
        License Priority: Medium


sh log -
*Aug 20 13:34:09.507: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Initialized
*Aug 20 13:34:09.511: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Enabled
*Aug 20 2016 06:34:16.203 MST: %VPN_HW-6-INFO_LOC: Crypto engine: slot 0  State changed to: Initialized
*Aug 20 2016 06:34:16.203 MST: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0  State changed to: Disabled
*Aug 20 2016 06:34:16.223 MST: %VPN_HW-6-INFO_LOC: Crypto engine: slot 0  State changed to: Enabled
*Aug 20 2016 06:34:16.223 MST:  ISM VPN UP & READY


sh diag -
Internal Services Module (ISM) Slot 0
        Internal Services Module - Crypto Engine
        Internal Services Module is analyzed
        Internal Services Module insertion time 00:04:05 ago
        EEPROM contents at hardware discovery:
        PCB Serial Number        : ***
        Hardware Revision        : 1.0
        Part Number              : 73-13342-05
        Top Assy. Part Number    : 800-37237-02
        Board Revision           : A0
        Deviation Number         : 0
        Fab Version              : 02
        Product (FRU) Number     : ISM-VPN-29
        Version Identifier       : V02
        CLEI Code                : CMUCACLBAA



sh cry eng conf -
   crypto engine name:  Virtual Private Network (VPN) Module
        crypto engine type:  hardware
                     State:  Enabled
                  Location:  slot 0
              Product Name:  ISM VPN Accelerator
              UBOOT Ver   : U-Boot 1.1.1 - ISRG2-Crypto-Engine - Version 2.7 (Build time: Mar  7 2011 - 09:12:23)
              Firmware Ver:   User: dranjit - View/Label: REVENTON_FW_COMMIT_IOS_07122012 - Date: Jul 12 2012 - Time: 18:28:58
              HW State    : READY


sh run -

crypto ipsec transform-set DMVPNtrans esp-des esp-md5-hmac
 mode transport
!
crypto ipsec profile ***
 set transform-set DMVPNtrans


interface Tunnel0
 description DMVPN
 bandwidth 3000
 ip address ***
 no ip redirects
 ip mtu 1400
 ip flow ingress
 ip flow egress
 ip nat inside
 ip nhrp authentication ***
 ip nhrp map multicast dynamic
 ip nhrp map multicast ***
 ip nhrp map ***
 ip nhrp network-id ***
 ip nhrp holdtime 300
 ip nhrp nhs ***
 ip virtual-reassembly in
 ip tcp adjust-mss 1360
 load-interval 30
 tunnel source GigabitEthernet0/2
 tunnel mode gre multipoint
 tunnel key ***
 tunnel protection ipsec profile ***


interface GigabitEthernet0/2
 description spoke isp
 bandwidth 200000
 ip address 2.2.2.2 255.255.255.240
 ip access-group edge-ingress in
 ip access-group edge-egress out
 ip flow ingress
 ip flow egress
 ip nat outside
 ip virtual-reassembly in
 duplex full
 speed 1000
 no cdp enable
!

interface GigabitEthernet0/0
 description spoke lan
 ip address 192.168.4.1 255.255.252.0
 ip flow ingress
 ip flow egress
 ip nat inside
 ip virtual-reassembly in
 ip policy route-map path-control
 duplex auto
 speed auto
!








download from hub to spoke before activating ISM-VPN card -


iperf -

Starting Test: protocol: TCP, 1 streams, 1048576 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  3.02 MBytes  25.3 Mbits/sec
[  4]   1.00-2.01   sec  3.50 MBytes  29.2 Mbits/sec
[  4]   2.01-3.00   sec  3.26 MBytes  27.5 Mbits/sec
[  4]   3.00-4.00   sec  3.20 MBytes  26.8 Mbits/sec
[  4]   4.00-5.00   sec  3.38 MBytes  28.4 Mbits/sec
[  4]   5.00-6.00   sec  3.24 MBytes  27.2 Mbits/sec
[  4]   6.00-7.00   sec  3.40 MBytes  28.5 Mbits/sec
[  4]   7.00-8.00   sec  3.39 MBytes  28.4 Mbits/sec
[  4]   8.00-9.00   sec  3.38 MBytes  28.4 Mbits/sec
[  4]   9.00-10.00  sec  3.28 MBytes  27.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  33.2 MBytes  27.8 Mbits/sec                  sender
[  4]   0.00-10.00  sec  33.2 MBytes  27.8 Mbits/sec                  receiver
CPU Utilization: local/receiver 5.6% (2.1%u/3.5%s), remote/sender 0.3% (0.1%u/0.1%s)


2911 cpu, each peak is a test -

     5555544444          44444555551111111111444445555533333
      911111333334444477777444441111188888222224444466666000003333
  100
   90
   80
   70
   60                                               *****
   50  *****                    *****               *****
   40  **********          **********          **********
   30  **********          **********          ***************
   20  **********          ***************     ***************
   10 ***********     ****************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)






download from hub to spoke after activating ISM-VPN card -


iperf

Starting Test: protocol: TCP, 1 streams, 1048576 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec  1.53 MBytes  12.7 Mbits/sec
[  4]   1.01-2.00   sec  1.72 MBytes  14.5 Mbits/sec
[  4]   2.00-3.01   sec   809 KBytes  6.58 Mbits/sec
[  4]   3.01-4.01   sec   550 KBytes  4.51 Mbits/sec
[  4]   4.01-5.00   sec  1.67 MBytes  14.1 Mbits/sec
[  4]   5.00-6.00   sec  1.99 MBytes  16.7 Mbits/sec
[  4]   6.00-7.01   sec  1.86 MBytes  15.4 Mbits/sec
[  4]   7.01-8.00   sec  1.89 MBytes  16.0 Mbits/sec
[  4]   8.00-9.00   sec  2.28 MBytes  19.2 Mbits/sec
[  4]   9.00-10.01  sec  1.97 MBytes  16.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec  16.3 MBytes  13.7 Mbits/sec                  sender
[  4]   0.00-10.01  sec  16.3 MBytes  13.7 Mbits/sec                  receiver
CPU Utilization: local/receiver 3.0% (1.6%u/1.4%s), remote/sender 0.0% (0.0%u/0.0%s)



each cpu peak is a throughput test -


      99999999999999999999999     222229999977777          9999999
      444444446666699999999998888844444777774444433333777773333300
  100         ***************          *****
   90 ***********************          *****               *******
   80 ***********************          *****               *******
   70 ***********************          **********          *******
   60 ***********************          **********          *******
   50 ***********************          **********          *******
   40 ***********************          **********          *******
   30 ***********************          **********          *******
   20 ***********************     ***************          *******
   10 *******************************************     ************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)






upload from spoke to hub before activating ISM-VPN card -

Starting Test: protocol: TCP, 1 streams, 1048576 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  5.30 MBytes  44.4 Mbits/sec
[  4]   1.00-2.00   sec  5.53 MBytes  46.4 Mbits/sec
[  4]   2.00-3.00   sec  5.66 MBytes  47.5 Mbits/sec
[  4]   3.00-4.00   sec  5.92 MBytes  49.7 Mbits/sec
[  4]   4.00-5.00   sec  5.68 MBytes  47.6 Mbits/sec
[  4]   5.00-6.00   sec  5.89 MBytes  49.4 Mbits/sec
[  4]   6.00-7.00   sec  6.01 MBytes  50.4 Mbits/sec
[  4]   7.00-8.00   sec  5.72 MBytes  48.0 Mbits/sec
[  4]   8.00-9.00   sec  5.88 MBytes  49.4 Mbits/sec
[  4]   9.00-10.00  sec  6.05 MBytes  50.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  57.9 MBytes  48.6 Mbits/sec                  sender
[  4]   0.00-10.00  sec  57.9 MBytes  48.6 Mbits/sec                  receiver
CPU Utilization: local/receiver 10.7% (4.3%u/6.4%s), remote/sender 0.8% (0.3%u/0.5%s)






upload from spoke to hub after activating ISM-VPN card:

Starting Test: protocol: TCP, 1 streams, 1048576 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  8.00 MBytes  67.0 Mbits/sec
[  4]   1.00-2.01   sec  8.00 MBytes  66.8 Mbits/sec
[  4]   2.01-3.01   sec  8.00 MBytes  67.1 Mbits/sec
[  4]   3.01-4.01   sec  8.00 MBytes  67.1 Mbits/sec
[  4]   4.01-5.00   sec  9.00 MBytes  75.8 Mbits/sec
[  4]   5.00-6.01   sec  8.00 MBytes  66.8 Mbits/sec
[  4]   6.01-7.01   sec  8.00 MBytes  67.1 Mbits/sec
[  4]   7.01-8.01   sec  8.00 MBytes  67.1 Mbits/sec
[  4]   8.01-9.01   sec  8.00 MBytes  67.1 Mbits/sec
[  4]   9.01-10.01  sec  9.00 MBytes  75.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec  82.0 MBytes  68.7 Mbits/sec                  sender
[  4]   0.00-10.01  sec  81.1 MBytes  67.9 Mbits/sec                  receiver
CPU Utilization: local/sender 0.9% (0.3%u/0.6%s), remote/receiver 1.5% (0.5%u/1.0%s)




      99999999933333
      555577777999993333355555444444444433333444445555533333333333
  100 *********
   90 *********
   80 *********
   70 *********
   60 *********
   50 *********
   40 **************
   30 **************
   20 **************
   10 **************     *****                    *****
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

only "IP Input" shows up using much cpu in the processes while pushing a test transfer:

upload from spoke to hub with ism-vpn enabled -
sh proc cpu sorted
CPU utilization for five seconds: 97%/72%; one minute: 40%; five minutes: 13%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 168       56308      137900        408 22.79%  8.92%  2.57%   0 IP Input


download from hub to spoke with ism-vpn enabled -
CPU utilization for five seconds: 84%/40%; one minute: 65%; five minutes: 29%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 168       99572      160143        621 40.23% 19.40%  7.32%   0 IP Input

also will note that with or without ism-vpn enabled there is slight improvement in throughput if ip policy route-map path-control which we use for 3 internet traffic routes unrelated to dmvpn is removed from spoke lan interface or ACL removed from internet interface, ingress spoke to hub throughput still remains poor.  it's understood that 2911 aggregate throughout performance will take a hit with acl & route-maps in use -

egress spoke to hub:

Starting Test: protocol: TCP, 1 streams, 1048576 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec  9.00 MBytes  75.0 Mbits/sec
[  4]   1.01-2.01   sec  10.0 MBytes  83.9 Mbits/sec
[  4]   2.01-3.01   sec  10.0 MBytes  83.8 Mbits/sec
[  4]   3.01-4.01   sec  10.0 MBytes  83.9 Mbits/sec
[  4]   4.01-5.01   sec  10.0 MBytes  83.9 Mbits/sec
[  4]   5.01-6.01   sec  2.00 MBytes  16.8 Mbits/sec
[  4]   6.01-7.01   sec  7.00 MBytes  58.7 Mbits/sec
[  4]   7.01-8.01   sec  9.00 MBytes  75.5 Mbits/sec
[  4]   8.01-9.01   sec  10.0 MBytes  83.9 Mbits/sec
[  4]   9.01-10.01  sec  10.0 MBytes  83.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec  87.0 MBytes  72.9 Mbits/sec                  sender
[  4]   0.00-10.01  sec  86.6 MBytes  72.6 Mbits/sec                  receiver
CPU Utilization: local/sender 1.1% (0.1%u/0.9%s), remote/receiver 2.3% (0.6%u/1.7%s)

ingress hub to spoke:

Starting Test: protocol: TCP, 1 streams, 1048576 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec  1.52 MBytes  12.7 Mbits/sec
[  4]   1.01-2.00   sec  1.54 MBytes  13.0 Mbits/sec
[  4]   2.00-3.00   sec  1.71 MBytes  14.4 Mbits/sec
[  4]   3.00-4.00   sec  1.94 MBytes  16.2 Mbits/sec
[  4]   4.00-5.00   sec  1.82 MBytes  15.3 Mbits/sec
[  4]   5.00-6.00   sec  2.51 MBytes  21.0 Mbits/sec
[  4]   6.00-7.01   sec  1.96 MBytes  16.3 Mbits/sec
[  4]   7.01-8.01   sec  1.95 MBytes  16.4 Mbits/sec
[  4]   8.01-9.00   sec  2.12 MBytes  17.9 Mbits/sec
[  4]   9.00-10.00  sec  1.98 MBytes  16.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  19.1 MBytes  16.0 Mbits/sec                  sender
[  4]   0.00-10.00  sec  19.1 MBytes  16.0 Mbits/sec                  receiver
CPU Utilization: local/receiver 4.4% (2.6%u/1.8%s), remote/sender 0.3% (0.1%u/0.2%s)

Labels:
0 Helpful
2 Replies 2

wayfaring
Level 1
Level 1

‎08-23-2016 06:50 AM

http://www.cisco.com/c/en/us/products/collateral/routers/3900-series-integrated-services-routers-isr/q-and-a-c67-606268.html

Q.    What is the HSEC-k9 license?
A.     The HSEC-K9 license removes the curtailment enforced by the U.S. government export restrictions on the encrypted tunnel count and encrypted throughput. HSEC-K9 is available only on the Cisco 2921, Cisco 2951, Cisco 3925, Cisco 3945, Cisco 3925E, and Cisco 3945E. With the HSEC-K9 license, the ISR G2 router can go over the curtailment limit of 225 tunnels maximum for IP Security (IPsec) and encrypted throughput of 85 -Mbps unidirectional traffic in or out of the ISR G2 router, with a bidirectional total of 170 Mbps. The Cisco 1941, 2901, and 2911 already have maximum encryption capacities within export limits.
.
.
How should this statement be interpreted for usage of the ISM-VPN-29 hardware addon on this 2911? 
.
.
edit: also spotted a couple of these yesterday but it may have resulted only from various udp & tcp iperf tests -
*Aug 22 2016 12:23:06.146 MST: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Tunnel0: the fragment table has reached its maximum threshold 16
*Aug 22 2016 11:42:37.574 MST: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Tunnel0: the fragment table has reached its maximum threshold 16

0 Helpful

wayfaring
Level 1
Level 1

‎01-06-2017 09:18 AM

Just checking back to see if there are any suggestions or experience with these before I toss it and move on to looking at other hardware.  2911's severe throughput limitations with acl + route maps in use are a known factor but performance overall actually seems worse since this card was enabled. Example - high latency occurs on pings from lan client to lan interface or ping from locally within router and is reflected in response of network resources sporadically when pushing SMB traffic across the dmvpn link at 10-25Mb with corresponding cpu spike until traffic subsides.

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card