Slow VPN throughput

cciesec2011 · ‎03-27-2011

I have site-2-site IPSec VPNs as follows:

- Site A to Site C,
- Site B to Site C,
- Site A is in Raleigh. Site B is New York. Site C is New Jersey
- Latency is 25msec between Site A & Site C, 27msec between Site B and Site C,
- Latency is 29msec between Site A and Site C.

- Each location is connected to the Internet with OC-3 speed.
- Site A has Cisco VXR 7206 router with VAM+ encryption card,
- Site B has Cisco ASR1002 with GE5 engine,
- Site C has Cisco 3845 with AIM card,

All routers are behind checkpoint firewalls (NGx R71.30). All location are connected
to OC-3 Internet connection at each location. Site A Internet provider is Internap.
Site B Internap provider is also Intenap. Site C Intenap provider is Level 3.

I can push 120Mbps IPSec traffics (AES-256/SHA, DH-5 with PFS Group 5)
between Site A and Site C without any issues,

I can push 125Mbps IPSec traffics (AES-256/SHA, DH-5 with PFS Group 5)
between Site B and Site C,

Last week, when I tried to bring up VPN between Site A and Site B, the VPN came up just
fine except that I can only push at most 8Mbps between Site A and Site C using Iperf.

All of these VPN devices sit behind the Checkpoint firewalls. The checkpoint just
routes the traffics and not doing any NAT for IPSec traffics. The firewalls have very low
utilization. I have rule out the Checkpoint firewalls as the source of the problem because
IPSec VPN between Site A <-> C and B <-> C can push 125Mbps of VPN traffics. Furthermore,
I also have hosts directly connected behind the Checkpoint firewalls and those hosts can
download/upload almost 150Mbps. I've tried different options with Iperf between site A <-> B
and the most I can get is 8Mbps.

At this point, I am at a loss here. If anyone can suggest something else I can try, that would
be great.

Thanks in advance.

Marcin Latosiewicz · ‎03-27-2011

Hi,

First of all you need to understand why the performance problem persist. Check sniffer trace of that iperf you're doing.

Check crypto accelerator stats and CEF drops.(If you want I can give you some commands but I need to understand the versions of software).

I understand that it's a VAM2+ and not VAM+?

BTW PFS is only done once during rekey, so it's not very relevant to performance.

Marcin

cciesec2011 · ‎03-27-2011

I already checked the trace from iperf since these hosts are linux hosts. looking at tcpdump, I am not seeing a lot re-transmissions which indicate the communication is clean:

3845: c3845-adventerprisek9-mz.124-24.T4.bin

ASR 1002: asr1000rp1-adventerprisek9.02.06.02.122-33.XNF2.bin

7206VXR: c7200-ik9o3s-mz.124-25c.bin

VPN-1#show cef drop
% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Drop Statistics
Slot Encap_fail Unresolved Unsupported No_route No_adj ChkSum_Err
RP 0 0 0 0 0 0
VPN-1#

where else should I look next?

Marcin Latosiewicz · ‎03-27-2011

Well I assume we're focusing on A and B?

IOS

"show crypto engine accel stati"

"show ip cef switch stati"

IOS XE

show platform hardware qfp active statistics drop | e _0_
sh pl ha qf ac fe ipsec data drop

This will give us some basic info. It's important taht you run those commands a few times while running the test.

Marcin

cciesec2011 · ‎03-27-2011

I will provide them tomorrow. By the way, does the command "show ip cef switch stat" does NOT exist on the IOS of the VXR 7206 I am running:

V0N>sh ip cef ?
   scanner            CEF scanner trigger statistics
summary            CEF table summary
   |                  Output modifiers

V0N>sh ip cef

I am curious. What kind of info are you looking for from these output? If you still can not determine the root cause, what would be the next step?

Marcin Latosiewicz · ‎03-28-2011

We need to establlish where/if the packets are dropped on data path:

- CEF

- buffers

- crypto accelerator stats

And how does the exchange look like - sniffer trace.

- You said it's not latency...

- You metion that performance is fine on devices just in front VPN routers, it's time to see what happens just after VPN routers.

Logical next step is to confirm whether those packets are dropped/delayed/OoO'ed by VPN routers.

There's a limited scope of what we can do via forums and the amount of info we can collect/analyze.

Marcin

Marcin Latosiewicz · ‎03-28-2011

By the way, does the command "show ip cef switch stat" does NOT exist on the IOS of the VXR 7206 I am running:

V0N>sh ip cef ?
   scanner            CEF scanner trigger statistics
summary            CEF table summary
   |                  Output modifiers

V0N>sh ip cef

Ah indeed you're running mainline ...

"show cef drop"

"show cef not-cef"

would be a place to start.

Marcin