I have a CSR 1000V in two separate AWS VPC's. I have an ipsec tunnel configured between them.
I have a linux host on each end and have been using iperf3 to test network performance.
If I specify one udp stream with iperf3 I can max out the throughput at around 300 Mbps. However if use one tcp stream I cannot get beyond 20 Mbps. If I increase the streams with tcp I can also max out the throughput but none of them will go beyond 20 Mbps individually.
I have messed with mtu/mss and tcp window sizes but no matter what I have changed I cannot get beyond 20 Mbps using one tcp stream. I have provided the current tunnel interface configs and encryption used on the tunnel below also.
I can get 400 Mbps using one tcp stream going directly over the internet not going through the tunnel so this does not make sense. Any help would be appreciated.
Examples:
One TCP Stream:
ubuntu@ip-192-168-200-124:~$ iperf3 -c 10.0.30.32 -b 0
Connecting to host 10.0.30.32, port 5201
[ 5] local 192.168.200.124 port 60468 connected to 10.0.30.32 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.03 MBytes 8.60 Mbits/sec 53 10.7 KBytes
[ 5] 1.00-2.00 sec 2.55 MBytes 21.4 Mbits/sec 144 8.03 KBytes
[ 5] 2.00-3.00 sec 1.71 MBytes 14.3 Mbits/sec 107 12.0 KBytes
[ 5] 3.00-4.00 sec 3.46 MBytes 29.0 Mbits/sec 121 2.68 KBytes
[ 5] 4.00-5.00 sec 1.15 MBytes 9.61 Mbits/sec 97 2.68 KBytes
[ 5] 5.00-6.00 sec 409 KBytes 3.35 Mbits/sec 28 2.68 KBytes
[ 5] 6.00-7.00 sec 1.01 MBytes 8.51 Mbits/sec 65 9.37 KBytes
[ 5] 7.00-8.00 sec 1.15 MBytes 9.62 Mbits/sec 61 2.68 KBytes
[ 5] 8.00-9.00 sec 1.17 MBytes 9.78 Mbits/sec 79 5.35 KBytes
[ 5] 9.00-10.00 sec 1.33 MBytes 11.1 Mbits/sec 89 2.68 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 14.9 MBytes 12.5 Mbits/sec 844 sender
[ 5] 0.00-10.04 sec 14.9 MBytes 12.4 Mbits/sec receiver
One UDP Stream:
ubuntu@ip-192-168-200-124:~$ iperf3 -c 10.0.30.32 -b 0 -u
Connecting to host 10.0.30.32, port 5201
[ 5] local 192.168.200.124 port 42720 connected to 10.0.30.32 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 36.8 MBytes 308 Mbits/sec 28130
[ 5] 1.00-2.00 sec 38.5 MBytes 323 Mbits/sec 29470
[ 5] 2.00-3.00 sec 42.1 MBytes 353 Mbits/sec 32250
[ 5] 3.00-4.00 sec 36.9 MBytes 310 Mbits/sec 28260
[ 5] 4.00-5.00 sec 36.2 MBytes 303 Mbits/sec 27670
[ 5] 5.00-6.00 sec 36.1 MBytes 303 Mbits/sec 27640
[ 5] 6.00-7.00 sec 37.4 MBytes 313 Mbits/sec 28600
[ 5] 7.00-8.00 sec 36.1 MBytes 303 Mbits/sec 27610
[ 5] 8.00-9.00 sec 36.4 MBytes 306 Mbits/sec 27890
[ 5] 9.00-10.00 sec 40.3 MBytes 338 Mbits/sec 30840
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 377 MBytes 316 Mbits/sec 0.000 ms 0/288360 (0%) sender
[ 5] 0.00-10.19 sec 338 MBytes 278 Mbits/sec 0.079 ms 29363/287976 (10%) receiver
Tunnel Interfaces:
interface Tunnel10
ip address 172.16.0.1 255.255.255.252
ip mtu 1422
ip tcp adjust-mss 1382
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination
tunnel path-mtu-discovery
tunnel protection ipsec profile IPSEC_PROFILE
interface Tunnel10
ip address 172.16.0.2 255.255.255.252
ip mtu 1422
ip tcp adjust-mss 1382
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination
tunnel path-mtu-discovery
tunnel protection ipsec profile IPSEC_PROFILE
csr-perf-1#sh crypto session detail
Crypto session current status
Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication, F - IKE Fragmentation
R - IKE Auto Reconnect, U - IKE Dynamic Route Update
S - SIP VPN
Interface: Tunnel10
Profile: IKEv2_PROFILE
Uptime: 00:26:16
Session status: UP-ACTIVE
Peer: 1.2.3.4 port 4500 fvrf: (none) ivrf: (none)
Phase1_id: 1.2.3.4
Desc: (none)
Session ID: 7
IKEv2 SA: local 10.0.100.234/4500 remote 1.2.3.4/4500 Active
Capabilities:DNU connid:1 lifetime:23:33:44
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 445403 drop 0 life (KB/Sec) 3981851/2024
Outbound: #pkts enc'ed 120091 drop 0 life (KB/Sec) 4601335/2024
csr-perf-1#sh crypto ipsec profile
IPSEC profile IPSEC_PROFILE
IKEv2 Profile: IKEv2_PROFILE
Security association lifetime: 4608000 kilobytes/3600 seconds
Responder-Only (Y/N): N
PFS (Y/N): N
Mixed-mode : Disabled
Transform sets={
IPSEC_TRANSFORM1: { esp-gcm 256 } ,
}
csr-perf-1#sh interfaces tunnel10
Tunnel10 is up, line protocol is up
Hardware is Tunnel
Internet address is 172.16.0.1/30
MTU 9938 bytes, BW 100 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 58/255, rxload 255/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel linestate evaluation up
Tunnel source 10.0.100.234 (GigabitEthernet1), destination 1.2.3.4
Tunnel Subblocks:
src-track:
Tunnel10 source tracking subblock associated with GigabitEthernet1
Set of tunnels with source GigabitEthernet1, 1 member (includes iterators), on interface <OK>
Tunnel protocol/transport IPSEC/IP
Tunnel TTL 255
Path MTU Discovery, ager 10 mins, min MTU 92
Tunnel transport MTU 1438 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Tunnel protection via IPSec (profile "IPSEC_PROFILE")
Last input 00:00:02, output 00:00:02, output hang never
Last clearing of "show interface" counters 05:57:38
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 1686055
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 2871000 bits/sec, 244 packets/sec
5 minute output rate 23000 bits/sec, 62 packets/sec
9038444 packets input, 12177968162 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
4452945 packets output, 4193319560 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
Physical Interface Tunnel is tied to:
interface GigabitEthernet1
ip address dhcp
ip nat outside
negotiation auto
no mop enabled
no mop sysid
