03-28-2009 01:25 PM
Need advise from someone with extensive experience on the Cisco 2851 platform.
LAN_X---CP-Firewall----Internet---2851---LAN_Y
Site-2-Site VPN betwen a Checkpoint NGx R70 firewall and a Cisco 2851 running IOS 12.4(24)T c2800nm-advipservicesk9-mz.124-24.T.bin. The Checkpoint firewall is capable of pushing 500Mbps
IPSec VPN AES-256/DH-5/PFS-5. VPN between the NGx R70 and Cisco 2851 is working but I can only
push 8Mbps when CPU on the Cisco 2851 reaches 98% CPU utilization:
R2851-3#sh process cpu | i five
CPU utilization for five seconds: 97%/17%; one minute: 97%; five minutes: 97%
R2851-3#sh process cpu | i five
CPU utilization for five seconds: 97%/17%; one minute: 97%; five minutes: 97%
R2851-3#sh process cpu | i five
CPU utilization for five seconds: 97%/17%; one minute: 97%; five minutes: 97%
R2851-3#sh process cpu | i five
CPU utilization for five seconds: 97%/17%; one minute: 97%; five minutes: 97%
R2851-3#
R2851-3#sh int g0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is MV96340 Ethernet, address is 000a.b802.d4c0 (bia 000a.b802.d4c0)
Internet address is 192.168.15.201/24
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 22/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is T
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1w0d
Input queue: 20/75/348/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 8628000 bits/sec, 1350 packets/sec
30 second output rate 492000 bits/sec, 451 packets/sec
64248736 packets input, 2494966466 bytes, 5 no buffer
Received 2603888 broadcasts, 0 runts, 0 giants, 244 throttles
1813 input errors, 0 CRC, 0 frame, 0 overrun, 1813 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
36434076 packets output, 2472485499 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
R2851-3#
Any ideas why I can push only 8Mbps VPN traffics on the 2851?
04-02-2009 08:39 PM
Did you enable hardware crypto engine?
Use "show ver" or/and "show diag" to find out what VPN module do you have on this router and then use "crypto engine ..." command to enable it.
04-02-2009 08:49 PM
Additional info from Data sheet.
If using VPN module:
The Cisco 2800 Series Module (AIM-VPN/SSL-2) can provide hardware-based IPSec encryption services of 30 and 90 Mbps in the Cisco 2801, 35 and 100 Mbps in the Cisco 2811, 90 and 125 Mbps in the Cisco 2821, and 100 and 150 Mbps in the Cisco 2851 (IPSec IMIX and 1400-byte packets).
04-03-2009 08:57 AM
The VPN module is enable. Anymore ideas?
R2851-3#sh crypto engine accelerator statistic
Device: AIM-VPN/EPII-PLUS
Location: AIM Slot: 0
Virtual Private Network (VPN) Module in slot : 0
Statistics for Hardware VPN Module since the last clear
of counters 1192084 seconds ago
97506087 packets in 97506087 packets out
90174632666 bytes in 90915789436 bytes out
81 paks/sec in 81 paks/sec out
605 Kbits/sec in 610 Kbits/sec out
58251169 packets decrypted 39254918 packets encrypted
87145415176 bytes before decrypt 3770374260 bytes encrypted
83726767962 bytes decrypted 6447864704 bytes after encrypt
0 packets decompressed 0 packets compressed
0 bytes before decomp 0 bytes before comp
0 bytes after decomp 0 bytes after comp
0 packets bypass decompr 0 packets bypass compres
0 bytes bypass decompres 0 bytes bypass compressi
0 packets not decompress 0 packets not compressed
0 bytes not decompressed 0 bytes not compressed
1.0:1 compression ratio 1.0:1 overall
119919 commands out 119919 commands acknowledged
Last 5 minutes:
700980 packets in 700980 packets out
2336 paks/sec in 2336 paks/sec out
17346343 bits/sec in 17501119 bits/sec out
608470038 bytes decrypted 14480060 bytes encrypted
16445136 Kbits/sec decrypted 391352 Kbits/sec encrypted
1.0:1 compression ratio 1.0:1 overall
HSP details:
hsp_operations : 119935 hsp_sessions : 4
R2851-3#
04-05-2009 10:45 AM
I am thinking if there is a fragmentation issue here. Since if haredware vpn engine is enabled, the router's CPU should not be that high. Please check the following link to see if it can be a help.
http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml
04-05-2009 04:19 PM
" the router's CPU should not be that high."
It should not but it is. I am not sure if the link you provided can resolve my issue since I am not using GRE/IPSec, just straight forward IPSec.
I can say that if I replace the Cisco 2851 with another Checkpoint firewall, I can easily push 500+ mbps IPSec traffics with IPerf. As soon as I put the Cisco 2851 back in place, I am stuck at 8Mbps throughput and that the CPU stays contanst at 96% CPU utilization.
04-05-2009 04:40 PM
Hi David, per data sheet, 2851 should be able to handle around 100M bps vpn traffic with vpn hardware module. If it is stuck at 8Mbps and have a high cpu, it looks like the packet was processed by CPU instead of VPN module for certain reason.
Can you try packet size 1400 byte when using iperf to do bandwidth testing?
04-05-2009 06:09 PM
well I did one step better. I set the MTU on both the Cisco 2851 and the Checkpoint firewall to 1400. I also run iperf at 1400 byte packet size. CPU is till 98% at between 8mbps and 9mbps. The bandwidth varies betwen 8mbps and 20mbps but CPU is constant at 99% utilization.
04-05-2009 06:12 PM
don't change MTU on cisco2851 and checkpoint. Keep them as default 1500. Just use 1400 byte on your iperf testing.
04-05-2009 06:37 PM
I am not sure what you're trying to achieve here but I changed the MTU on both Checkpoint and Cisco back to 1500 and 1400 bytes on my iperf. Same issue, CPU on 2851 hits 99% utilization at 8mbps.
I even set the mtu on my linux client to 1400. Furthermore, I have full MTU path discovery end-to-end but no luck.
anymore ideas?
04-06-2009 08:25 AM
Performance testing in data sheet is done by using 1400 bytes packet size. If 1400 bytes packet size is used and all interface MTU on the path is 1500 bytes, packet should not be fragmentated.
Based on your testing, it does not look like a fragmentation issue here.
By the way, which process uses most CPU? Can you post "show process cpu sort"?
04-06-2009 09:53 AM
R2851-3#sh process cpu sorted
CPU utilization for five seconds: 97%/16%; one minute: 97%; five minutes: 72%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
220 15115924 5603399 2697 34.47% 33.98% 19.34% 0 Crypto Support
6 11381916 3267358 3483 25.11% 24.76% 14.05% 0 Pool Manager
119 48853064 26696407 1829 17.83% 18.10% 18.08% 0 IP Input
252 4201672 8522276 493 2.23% 2.23% 2.11% 0 Crypto PAS Proc
184 404544 353832475 1 0.55% 0.53% 0.45% 0 HQF Shaper Backg
305 524 520 1007 0.47% 0.07% 0.05% 514 Virtual Exec
19 1332452 7192843 185 0.31% 0.28% 0.26% 0 ARP Input
18 2096 48605 43 0.07% 0.00% 0.00% 0 Environmental mo
185 23888 14560169 1 0.07% 0.03% 0.02% 0 RBSCP Background
192 3444 2846848 1 0.07% 0.00% 0.00% 0 Inspect process
37 10656 770731 13 0.07% 0.01% 0.00% 0 Net Background
04-06-2009 11:31 AM
Hi David, I could not think of anything else. I would like to suggest you to open a TAC case for further assistance.
I am not sure if VPN module is bad. If you want, you can try a new one.
Capture "show ip traffic" and "show buffer" before and after you testing to see if there is anything suspected.
04-06-2009 12:25 PM
Hi,
Many thanks for your feedback. I will open a TAC case in a few days.
For what it worths, I replaced this router with another identical 2851 but still run into the same issue.
Thanks again.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide