cisco 2811 with AIM-SSL2 produces 100% CPU load at only 4 Mbit/s

guilty_2 · ‎01-16-2012

Hi,

We are using a few Cisco2811's with the AIM-SSL2 hardware encryption module in a private WAN.

This WAN is ethernet based and uses private address space.

To add security, we are using cisco group-VPN with AES-128 encryption. The IOS version is advanced ip services.

Routing is done with OSPF.

According several sources on the cisco site this hardware combination should be able to handle a throughput of 30 -90 Mbit/s.

In effect, we see that the CPU uses 60% at 2 Mbit/s load and almost flatlines at 100% on 4 Mbit/s load.

With a 'show proc cpu' however the biggest individual process uses only 1,5%.

The testscenario at which the 30 - 90 Mbit/s load get reached is a point-to-point Ipsec solution encrypted at 3DES.

Questions:

- Has anyone in the field reached better performance in the scenario we utilize ?

- Did we select the proper hardware or are we using underspecced equipment ? If so, what router should we use to

reache at least a througput of 20 mb/s ?

Peter Paluch · ‎01-16-2012

Hello Rob,

This looks as if the router was not using the AIM module at all, or had significant work to do even if the AIM was used and offloaded the crypto operations. The throughput as indicated in datasheets is under ideal conditions but certainly, what you are experiencing is something deserving attention.

One thing to verify is the output of show crypto engine brief and show crypto engine accelerator statistic commands to verify if the AIM module is recognized, enabled and used (the show inventory and show diag may also be useful here although it will only print out the presence of the module in your router but not its usage state).

Another place to check is the running-config for any crypto engine commands - particularly if they are in the "no" form. These commands may deactivate the AIM module and force the crypto operations to be performed by the CPU itself.

Yet another place to look into is the show processes cpu sorted under high traffic load to see which process is consuming the most CPU cycles. That may indicate further the possible cause.

A thing immediately coming to my mind is the MTU issue - IPsec necessarily causes the packets to bloat because of additional encapsulation. If the resulting packets are too big, they need to be fragmented, and that can cause the CPU to spike up. I strongly recommend reading the following document carefully and handling the MTU accordingly:

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

Best regards,

Peter

guilty_2 · ‎01-16-2012

Hi Peter,

thanks for your quick reply. Here are some highlights of the proposed commands.

show crypto engine brief
        crypto engine name: Virtual Private Network (VPN) Module
        crypto engine type: hardware
        State: Enabled
        Location: aim 0
        VPN Module in slot: 0
        Product Name: AIM-VPN/SSL-2

show crypto engine accelerator statistic

Last 5 minutes:
       933793 packets in                      933793 packets out
          3112 paks/sec in                       3112 paks/sec out
      2982903 bits/sec in                    2961855 bits/sec out
      48599582 bytes decrypted               39232809 bytes encrypted
       1313502 Kbits/sec decrypted            1060346 Kbits/sec encrypted
      1.0:1 compression ratio                1.0:1 overall

Errors:
ppq full errors : 506 ppq rx errors : 0
NR overflow : 0 pkts dropped : 506

show inventory

NAME: "2811 chassis", DESCR: "2811 chassis"
PID: CISCO2811 , VID: V06 , SN: FHK124xxx

NAME: "Virtual Private Network (VPN) Module on Slot 0", DESCR: "Encryption AIM Element"
PID: AIM-VPN/SSL-2 , VID: V01, SN: FOC1241xxx

show proc cpu sorted:

CPU utilization for five seconds: 73%/70%; one minute: 73%; five minutes: 70%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
178     6750736 122200048         55 1.59% 1.45% 1.52%   0 HQF Shaper Backg
111      638552    554465       1151 0.23% 0.12% 0.09%   0 IP Input
   2      237820    102147       2328 0.07% 0.13% 0.13%   0 Load Meter
152       38556     15189       2538 0.07% 0.00% 0.00%   0 TCP Protocols
33       36080    510730         70 0.07% 0.00% 0.00%   0 GraphIt
19      229612    382633        600 0.07% 0.04% 0.03%   0 ARP Input

I'll also take a look into the MTU link you send me.

Thanks.

guilty_2 · ‎01-16-2012

Hi Kishore,

the 'testscenario' is the scenario CISCO claims 30-90 Mbit/s throughput is reached. I personally did not verify that

as i am not in a position to change the crypto type in a production situation.

this is the link i am referring to:

http://www.cisco.com/en/US/prod/collateral/routers/ps5853/data_sheet_vpn_aim_for_18128003800routers_ps5853_Products_Data_Sheet.html

I am using cisco group VPN, no GRE involved.

Peter Paluch · ‎01-16-2012

Rob,

Thank for posting the information. Your router seems to be spending most of its time handling interrupts which would suggest that no ordinary process switching or process-based packet handling is taking place which is good. However, spending 70% in interrupt context while handling just a few Mbps is still unusually high. The HQF shaper load is completely negligible here.

The counters of encrypted/decrypted traffic in Kbit/s are insanely high... They would suggest that the encryption/decryption speed was roughly at 1Gb/s. I can't see how that can be true - consider clearing the counters using the clear crypto engine accelerator counter command.

Are you running CEF? Please verify that using the show ip cef command. Also please make sure all your interfaces are using CEF - they should not contain any no ip route-cache comands.

Are there any other applications or services running on the router - IP Inspect, ZBFW, TCP Intercept, NAT, IPS/IDS? What IOS version are you using?

Also, has it actually been proven that it is the crypto traffic that causes this load?

Best regards,

Peter

guilty_2 · ‎01-16-2012

Hi Peter,

i agree the counter is insanely high, might be a bug in the output format.

Looks like there is a factor 1000 difference. So it might have to be bits/sec instead of Kbits/sec.

After clearing the accelerator counters:

Last 5 minutes:

101298 packets in 101298 packets out

2813 paks/sec in 2813 paks/sec out

3584908 bits/sec in 3590411 bits/sec out

7878416 bytes decrypted 4067189 bytes encrypted

1969604 Kbits/sec decrypted 1016797 Kbits/sec encrypted

1.0:1 compression ratio 1.0:1 overall

so no real difference there

sh int fast 0/0 | incl input rate|output rate

30 second input rate 980000 bits/sec, 1696 packets/sec

30 second output rate 2286000 bits/sec, 1669 packets/sec

sh int fast 0/1 | incl input rate|output rate

30 second input rate 3309000 bits/sec, 1642 packets/sec

30 second output rate 2057000 bits/sec, 1680 packets/sec

show ip cef (some octets censored using x's)

Prefix Next Hop Interface

0.0.0.0/0 192.168.xxx.254 FastEthernet0/1.400

0.0.0.0/8 drop

0.0.0.0/32 receive

10.xx.50.0/24 192.168.xxx.254 FastEthernet0/1.400

10.xx.51.0/24 192.168.xxx.254 FastEthernet0/1.400

sh ip int | incl Fast|route

FastEthernet0/0 is up, line protocol is up

IP route-cache flags are Fast, CEF

FastEthernet0/0.12 is up, line protocol is up

IP route-cache flags are Fast, CEF

FastEthernet0/1 is up, line protocol is up

FastEthernet0/1.400 is up, line protocol is up

IP route-cache flags are Fast, CEF

FastEthernet0/1.410 is up, line protocol is up

IP route-cache flags are Fast, CEF

FastEthernet0/1.420 is up, line protocol is up

IP route-cache flags are Fast, CEF

FastEthernet0/1.430 is up, line protocol is up

IP route-cache flags are Fast, CEF

I am not using any of the services you listed.

sh ver

Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(22)T, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 10-Oct-08 00:05 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

I am not sure i understand the question if it has been proven the crypto traffic is causing the load.

The only traffic passing this box is traffic going in unencrypted and leaving encrypted. There is some OSPF

running on this box, but as it is a relatively small WAN (about 10 networks) i do not expect OSPF to

be the troublemaker. That also does not show in the CPU load. Most anoying thing is actually that the

show proc cpu output only shows the high load, but not the actual proces causing that load, leaving me with

a big incertainty what is happing on a process level on this router.

Peter Paluch · ‎01-16-2012

Hi Rob,

Most anoying thing is actually that the show proc cpu output only shows the high load, but not the actual proces causing that load, leaving me with a big incertainty what is happing on a process level on this router.

Well, the catch here is that the high load caused by the CPU spending lots of time in the interrupt context. It is not process-based load. From the total load of 73%, only 3% are consumed by processes and the remaining 70% are caused by the router performing some interrupt-based work. The show processes cpu is therefore unable to show any particular process causing the high load because it is in fact no process at all - rather, it is an interrupt handler.

Personally, I suggest (re)visiting the following two documents:

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

If my time permits, I can try to set up a simple lab with a couple of 2811 routers and GETVPN to see if I can confirm similar results. However, I do not have AIM modules available, just on-board crypto engines - but that should mean that my performance should not be better than yours, and if it is, there is something VERY wicked going on.

Has the same phenomenon with high CPU been observed on other routers in your VPN?

Best regards,

Peter

guilty_2 · ‎01-17-2012

Hi Peter,

"Has the same phenomenon with high CPU been observed on other routers in your VPN?"

Yes, all routers in the WAN (and in the same groupVPN) show exact the same behaviour.

Kishore Chennupati · ‎01-16-2012

Rob,

The testscenario at which the 30 - 90 Mbit/s load get reached is a point-to-point Ipsec solution encrypted at 3DES.

so let me ask you this. You do get the required throughput when you use 3DES, its only when you use AES that you see the spike .would i be right? As peter mentioned the performance switching on the 2811 should be enough for your requirements. Have you checked if the IOS has any bugs in terms of using IPsec with AES etc. Is it possible to downgrade or upgrade your IOS on just one device and see if it helps.

Also as peter recommended cheking MTU is also a good thing. Are you using GRE as well or just IPsec?

HTH

Kishore

guilty_2 · ‎01-16-2012

Hi Kishore,

the 'testscenario' is the scenario CISCO claims 30-90 Mbit/s throughput is reached. I personally did not verify that

as i am not in a position to change the crypto type in a production situation.

this is the link i am referring to:

http://www.cisco.com/en/US/prod/collateral/routers/ps5853/data_sheet_vpn_aim_for_18128003800routers_ps5853_Products_Data_Sheet.html

I am using cisco group VPN, no GRE involved.

guilty_2 · ‎01-17-2012

Hi Peter,

"Has the same phenomenon with high CPU been observed on other routers in your VPN?"

Yes, all routers in the WAN (and in the same groupVPN) show exact the same behaviour.

Peter Paluch · ‎01-17-2012

Rob,

So I've made a couple of experiments with 2811 routers running 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 15.1(4)M2 IOS. There were no AIM modules in these routers, only onboard VPN modules:

GM-1#show crypto engine brief

crypto engine name: Virtual Private Network (VPN) Module

crypto engine type: hardware

State: Enabled

Location: onboard 0

Product Name: Onboard-VPN

Middleware Version: v1.3.3

Firmware Version: v2.3.3

Time running: 6490 seconds

Compression: Yes

DES: Yes

3 DES: Yes

AES CBC: Yes (128,192,256)

AES CNTR: No

Maximum buffer length: 4096

Maximum DH index: 0000

Maximum SA index: 0000

Maximum Flow index: 2400

Maximum RSA key size: 2048

crypto engine name: Cisco VPN Software Implementation

crypto engine type: software

serial number: EADF0262

crypto engine state: installed

crypto engine in slot: N/A

GM-1#

Thy Key Server router relevant configuration:

crypto isakmp policy 1

encr aes

authentication pre-share

group 5

crypto isakmp key asdf address 192.0.2.1

crypto isakmp key asdf address 192.0.2.2

crypto isakmp key asdf address 192.0.2.3

!

crypto ipsec transform-set ESP-AES-SHA esp-aes esp-sha-hmac

!

crypto ipsec profile GETVPN-IPsec-Profile

set transform-set ESP-AES-SHA

!

crypto gdoi group B301

identity number 1

server local

rekey lifetime seconds 3600

rekey retransmit 60 number 3

rekey authentication mypubkey rsa GETVPN

rekey transport unicast

sa ipsec 1

profile GETVPN-IPsec-Profile

match address ipv4 100

replay counter window-size 64

address ipv4 192.0.2.4

!

interface Loopback0

ip address 192.0.2.4 255.255.255.255

!

access-list 100 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255

Relevant configuration of two Group Member "X" ("X" being 1, 2, 3):

crypto isakmp policy 1

encr aes

authentication pre-share

group 5

crypto isakmp key asdf address 192.0.2.4

!

crypto gdoi group B301

identity number 1

server address ipv4 192.0.2.4

client registration interface Loopback0

!

crypto map CM-fa0/0 1 gdoi

set group B301

!

interface Loopback0

ip address 192.0.2.X 255.255.255.255 !

interface FastEthernet0/0

ip address 10.1.X.2 255.255.255.0

duplex auto

speed auto

crypto map CM-fa0/0

!

interface FastEthernet0/1

ip address 10.0.X.1 255.255.255.0

duplex auto

speed auto

On Fa0/1 interfaces, I had PCs connected running D-ITG as traffic generator. I have been generating one-way UDP flows between PCs 10.0.1.2 and 10.0.2.2, each connected to its own GM router, running it through the GETVPN cloud.

The results were rather surprising:

200 pps, 1428 Bpp, roughly 2.2Mbps: sustained CPU load 14%/13%

400 pps, 1428 Bpp, roughly 4.5Mbps: sustained CPU load 26%/25%

600 pps, 1428 Bpp, roughly 6.8Mbps: sustained CPU load 39%/37%

800 pps, 1428 Bpp, roughly 9.1Mbps: sustained CPU load 50%/48%

1000 pps, 1428 Bpp, roughly 11.4Mbps: sustained CPU load 63%/60%

1200 pps, 1428 Bpp, roughly 13.7Mbps: sustained CPU load 76%/72%

1600 pps, 714 Bpp, roughly 9.1Mbps: sustained CPU load 61%/57%

3200 pps, 357 Bpp, roughly 9.1Mbps: sustained CPU load 93%/92%

(pps = packets per second, Bpp = bytes per packet)

As you can see, I was easily able to go up to 13.7Mbps with large packets quite safely. The 1428-byte IP packets were even fragmented before encryption (I have captured the internal GETVPN traffic and saw ESP packets of alternating sizes slightly above 1300 and 300 bytes).

Naturally, producing smaller packets more often yielded higher CPU loads - compare the last two lines to the 800pps/1428Bpp load at the same throughput.

However, using just plain 2811, I was able to reach much higher throughputs with less CPU load... Your issue really needs further investigation.

Do you believe you could post the entire configuration of the router experiencing the high loads?

Best regards,

Peter

guilty_2 · ‎01-17-2012

Hi Peter,

passwords and some ip addresses have been anonymized, but this is the whole config.

version 12.4

no service pad

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

!

hostname xxx

!

boot-start-marker

boot-end-marker

!

logging message-counter syslog

logging buffered 20000

no logging console

!

no aaa new-model

clock timezone CET 1

clock summer-time CET recurring last Sun Mar 2:00 last Sun Oct 2:00

!

dot11 syslog

ip source-route

!

ip cef

!

ip domain name yourdomain.com

ip vrf EXAMPLE1

rd 310:310

route-target export 310:1000

route-target import 310:1000

!

ip vrf EXAMPLE2

rd 300:300

route-target export 300:1000

route-target import 300:1000

!

ip vrf EXAMPLE3

rd 280:280

route-target export 280:1000

route-target import 280:1000

!

no ipv6 cef

!

multilink bundle-name authenticated

!

voice-card 0

!

crypto isakmp policy 10

encr aes

authentication pre-share

group 2

crypto isakmp key somekey1 address 192.168.0.1

crypto isakmp key somekey2 address 192.168.0.2

!

crypto gdoi group getvpn

identity number 1234

server address ipv4 192.168.0.1

server address ipv4 192.168.0.2

!

crypto map getvpn-map 10 gdoi

set group getvpn

!

archive

log config

hidekeys

!

interface FastEthernet0/0

description Inside interface

ip address 192.168.252.11 255.255.255.0

ip tcp adjust-mss 1360

load-interval 30

duplex full

speed 100

!

interface FastEthernet0/0.12

encapsulation dot1Q 12

ip vrf forwarding EXAMPLE2

ip address 192.168.148.201 255.255.255.0

!

interface FastEthernet0/1

description Outside interface to PE

no ip address

load-interval 30

duplex full

speed 100

!

interface FastEthernet0/1.400

description Outside interface to PE

encapsulation dot1Q 400

ip address 192.168.251.11 255.255.255.0

ip access-group 161 out

crypto map getvpn-map

!

interface FastEthernet0/1.410

encapsulation dot1Q 410

ip vrf forwarding EXAMPLE2

ip address 192.168.249.2 255.255.255.248

!

interface FastEthernet0/1.420

encapsulation dot1Q 420

ip vrf forwarding EXAMPLE1

ip address 192.168.249.10 255.255.255.248

!

interface FastEthernet0/1.430

encapsulation dot1Q 430

ip vrf forwarding EXAMPLE3

ip address 192.168.249.18 255.255.255.248

!

router ospf 100

log-adjacency-changes

redistribute static subnets route-map static_routes

network 192.168.251.0 0.0.0.255 area 0

network 192.168.252.0 0.0.0.255 area 0

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 192.168.251.254

ip route 172.25.0.0 255.255.0.0 192.168.252.2

ip route vrf EXAMPLE1 192.168.250.8 255.255.255.248 192.168.249.9

ip route vrf EXAMPLE2 192.168.250.0 255.255.255.248 192.168.249.1

ip route vrf EXAMPLE3 192.168.250.16 255.255.255.248 192.168.249.17

no ip http server

no ip http secure-server

ip http timeout-policy idle 60 life 86400 requests 10000

!

logging trap warnings

access-list 52 permit any

access-list 99 permit 172.25.x.60

access-list 99 permit 172.25.x.61

access-list 161 remark ACL at bootup router is fail closed until policy is loaded-no leak

access-list 161 permit esp any any

access-list 161 permit udp any eq isakmp any eq isakmp

access-list 161 permit udp any any eq isakmp

access-list 161 permit udp any any eq 848

access-list 161 permit ospf any any

access-list 161 deny ip any any

!

route-map static_routes permit 10

match ip address 52

!

control-plane

!

ccm-manager fax protocol cisco

!

mgcp fax t38 ecm

!

line con 0

session-timeout 10

exec-timeout 15 0

password xxx

logging synchronous

login

line aux 0

session-timeout 10

exec-timeout 5 0

password xxx

logging synchronous

login

line vty 0 4

session-timeout 60

exec-timeout 15 0

privilege level 15

password xxx

logging synchronous

login

transport input telnet ssh

line vty 5 15

privilege level 15

password xxx

logging synchronous

login

transport input telnet ssh

!

scheduler allocate 20000 1000

ntp access-group peer 99

ntp access-group query-only 99

ntp server 172.25.x.60

ntp server 172.25.x.61

end

Peter Paluch · ‎01-17-2012

Hello Rob,

Thanks for the config! Indeed, your configuration is rather basic - there is nothing special in there. Do you believe you could also post the output of the show crypto gdoi command?

On another note, what is the general nature of the traffic in your GETVPN - is it mostly data transfer or VoIP? I am trying to understand the nature of the flows.

Do you have a TAC contract? We are slowly reaching the limits of what we can try out.

Best regards,

Peter

guilty_2 · ‎01-17-2012

Hi Peter,

output of show crypto gdoi

GROUP INFORMATION

Group Name : getvpn

Group Identity : 1234

Rekeys received : 101

IPSec SA Direction : Both

Active Group Server : 192.168.0.2

Group Server list : 192.168.0.2

192.168.0.1

GM Reregisters in : 1271 secs

Rekey Received(hh:mm:ss) : 01:32:52

Rekeys received

Cumulative : 101

After registration : 101

Rekey Acks sent : 101

ACL Downloaded From KS 192.168.0.2:

access-list deny esp any any

access-list deny udp any port = 500 any port = 500

access-list deny udp any any port = 500

access-list deny udp any any port = 848

access-list deny ospf any any

access-list permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255

access-list permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255

access-list permit ip 172.0.0.0 0.255.255.255 172.0.0.0 0.255.255.255

access-list permit ip 193.0.0.0 0.255.255.255 172.0.0.0 0.255.255.255

access-list permit ip 194.0.0.0 0.255.255.255 172.0.0.0 0.255.255.255

access-list permit ip any any

KEK POLICY:

Rekey Transport Type : Unicast

Lifetime (secs) : 32789

Encrypt Algorithm : 3DES

Key Size : 192

Sig Hash Algorithm : HMAC_AUTH_SHA

Sig Key Length (bits) : 1024

TEK POLICY:

FastEthernet0/1.400:

IPsec SA:

sa direction:inbound

spi: 0x45F31DC(73347548)

transform: esp-aes esp-sha-hmac

sa timing:remaining key lifetime (sec): (1627)