dmvpn: one way throuput problem

kapp-werk · ‎03-16-2022

dmpvnHi,

we have a runnning dual hub dmvpn (2x ISR 4551 as HUB) ISR 4351 Router as Spoke.

all sites have fiber access. Central 600/600 Mbit. Spooke 300/300Mbit or 500/150Mbit.

we test with default iperf3 (-s / -c / -R) settings

upload from the spokes to central hit nearly the given upload speed.

but: from central to spoke me get between 70 - 20 Mbit and we can't find out where the problem is.

Rob Ingram · ‎03-16-2022

@kapp-werk does the speed fluctuate between 20-70Mbits? Do you have the HSEC license? Without this license traffic will be policed, which is why you may see the receive speed fluctuate.

Check the logs, you should receive a notification if you've exceeded Rx bandwidth.

Try with parallel streams use the "-P <number>" command option.

kapp-werk · ‎03-16-2022

Hi,

we try "iperf3 -c 192.168.100.XX -P 10" and get

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 23.6 MBytes 19.6 Mbits/sec 136 sender
[ 5] 0.00-10.00 sec 23.2 MBytes 19.5 Mbits/sec receiver
[ 7] 0.00-10.06 sec 22.2 MBytes 18.5 Mbits/sec 64 sender
[ 7] 0.00-10.00 sec 22.0 MBytes 18.4 Mbits/sec receiver
[ 9] 0.00-10.06 sec 21.5 MBytes 17.9 Mbits/sec 91 sender
[ 9] 0.00-10.00 sec 21.2 MBytes 17.8 Mbits/sec receiver
[ 11] 0.00-10.06 sec 16.8 MBytes 14.0 Mbits/sec 91 sender
[ 11] 0.00-10.00 sec 16.6 MBytes 13.9 Mbits/sec receiver
[ 13] 0.00-10.06 sec 23.6 MBytes 19.7 Mbits/sec 112 sender
[ 13] 0.00-10.00 sec 23.3 MBytes 19.6 Mbits/sec receiver
[ 15] 0.00-10.06 sec 30.9 MBytes 25.7 Mbits/sec 113 sender
[ 15] 0.00-10.00 sec 30.4 MBytes 25.5 Mbits/sec receiver
[ 17] 0.00-10.06 sec 27.4 MBytes 22.9 Mbits/sec 92 sender
[ 17] 0.00-10.00 sec 27.1 MBytes 22.7 Mbits/sec receiver
[ 19] 0.00-10.06 sec 23.3 MBytes 19.4 Mbits/sec 143 sender
[ 19] 0.00-10.00 sec 23.1 MBytes 19.4 Mbits/sec receiver
[ 21] 0.00-10.06 sec 21.5 MBytes 17.9 Mbits/sec 99 sender
[ 21] 0.00-10.00 sec 21.2 MBytes 17.8 Mbits/sec receiver
[ 23] 0.00-10.06 sec 19.0 MBytes 15.8 Mbits/sec 80 sender
[ 23] 0.00-10.00 sec 18.8 MBytes 15.7 Mbits/sec receiver
[SUM] 0.00-10.06 sec 230 MBytes 192 Mbits/sec 1021 sender
[SUM] 0.00-10.00 sec 227 MBytes 190 Mbits/sec receiver

This hit the limit on 4351.

iperf3 -c 192.168.100.44 -R
Connecting to host 192.168.100.44, port 5201
Reverse mode, remote host 192.168.100.44 is sending
[ 5] local 10.44.206.199 port 53335 connected to 192.168.100.44 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 7.14 MBytes 59.9 Mbits/sec
[ 5] 1.00-2.00 sec 9.24 MBytes 77.5 Mbits/sec
[ 5] 2.00-3.00 sec 7.96 MBytes 66.8 Mbits/sec
[ 5] 3.00-4.00 sec 10.2 MBytes 85.4 Mbits/sec
[ 5] 4.00-5.00 sec 6.88 MBytes 57.7 Mbits/sec
[ 5] 5.00-6.00 sec 8.64 MBytes 72.5 Mbits/sec
[ 5] 6.00-7.00 sec 6.68 MBytes 56.0 Mbits/sec
[ 5] 7.00-8.00 sec 7.56 MBytes 63.4 Mbits/sec
[ 5] 8.00-9.00 sec 6.75 MBytes 56.6 Mbits/sec
[ 5] 9.00-10.00 sec 7.98 MBytes 66.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.05 sec 79.7 MBytes 66.5 Mbits/sec 183 sender
[ 5] 0.00-10.00 sec 79.0 MBytes 66.3 Mbits/sec receiver

kapp-werk · ‎03-16-2022

sh lic all on hub

--

License Usage
==============

ISR_4400_UnifiedCommunication (ISR_4400_UnifiedCommunication):
Description: Unified Communications License for Cisco ISR 4400 Series
Count: 1
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED

ISR_4400_Security (ISR_4400_Security):
Description: Security License for Cisco ISR 4400 Series
Count: 1
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED

ISR_4451_2G_Performance (ISR_4451_2G_Performance):
Description: Performance on Demand License for 4450 Series
Count: 1
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED

ISR_4400_Hsec (ISR_4400_Hsec):
Description: U.S. Export Restriction Compliance license for 4400 series
Count: 1
Version: 1.0
Status: AUTHORIZED
Export status: RESTRICTED - ALLOWED
Feature Name: hseck9
Feature Description: Export Controlled Feature hseck9

sh lic all on spoke

--

License Usage
==============

ISR_4351_Security (ISR_4351_Security):
Description: Security License for Cisco ISR 4350 Series
Count: 1
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED

ISR_4351_Hsec (ISR_4351_Hsec):
Description: U.S. Export Restriction Compliance license for 4350 series
Count: 1
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED

MHM Cisco World · ‎03-16-2022

can you check the MTU for Hub and Spoke ?

kapp-werk · ‎03-17-2022

HUB:

Tun

interface Tunnel0
ip mtu 1300
ip tcp adjust-mss 1260
delay 1000
tunnel source GigabitEthernet0/0/2
tunnel path-mtu-discovery
end

SPOKE:

interface Tunnel1
ip mtu 1300
ip tcp adjust-mss 1260
tunnel source Dialer0
tunnel path-mtu-discovery
end

interface Dialer0
ip mtu 1452
encapsulation ppp

end

interface GigabitEthernet0/0/0.7
encapsulation dot1Q 7
pppoe enable group global
pppoe-client dial-pool-number 1
end

MHM Cisco World · ‎03-17-2022

Many MTU
discovery
under source interface
under tunnel interface

so to be check please do ping with df bit set and must the MTU as show with
show interface tunnel

if the ping success that OK if not then the Hub do more CPU work to frag and defrag the packet send through tunnel and hence the BW reduce.

kapp-werk · ‎03-17-2022

Hi,

we try:

spoke: #sh int tun 1
Tunnel1 is up, line protocol is up
Hardware is Tunnel
Description: Host dynamic tunnel
Internet address is 10.1.1.68/24
MTU 9918 bytes, BW 102400 Kbit/sec, DLY 10000 usec,
reliability 255/255, txload 1/255, rxload 10/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel linestate evaluation up
Tunnel source X.X.X.X (Dialer0), destination X.X.X.X
Tunnel Subblocks:
src-track:
Tunnel1 source tracking subblock associated with Dialer0
Set of tunnels with source Dialer0, 2 members (includes iterators), on interface <OK>
Tunnel protocol/transport GRE/IP
Key 0x186A0, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Path MTU Discovery, ager 10 mins, min MTU 92, MTU 1370, expires 00:09:20
Tunnel transport MTU 1370 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Tunnel protection via IPSec (profile "XXX")
Last input 00:00:00, output 00:00:03, output hang never
Last clearing of "show interface" counters 02:45:29
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo (QOS pre-classification)
Output queue: 0/0 (size/max)
5 minute input rate 4078000 bits/sec, 827 packets/sec
5 minute output rate 221000 bits/sec, 127 packets/sec
10761298 packets input, 8058830466 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
4561659 packets output, 2190390703 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

hub: #sh int tun 0

Tunnel0 is up, line protocol is up
Hardware is Tunnel
Description: Multi-Point GRE Tunnel for External
Internet address is 10.1.1.1/24
MTU 9972 bytes, BW 512000 Kbit/sec, DLY 10000 usec,
reliability 255/255, txload 29/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel linestate evaluation up
Tunnel source X.X.X.X (GigabitEthernet0/0/2)
Tunnel Subblocks:
src-track:
Tunnel0 source tracking subblock associated with GigabitEthernet0/0/2
Set of tunnels with source GigabitEthernet0/0/2, 1 member (includes iterators), on interface <OK>
Tunnel protocol/transport multi-GRE/IP
Key 0x186A0, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Path MTU Discovery, ager 10 mins, min MTU 92
Tunnel transport MTU 1472 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Tunnel protection via IPSec (profile "XXXX")
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 23:38:15
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 3249000 bits/sec, 2153 packets/sec
5 minute output rate 59179000 bits/sec, 11832 packets/sec
216098066 packets input, 56443667174 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
729573451 packets output, 449399411805 bytes, 0 underruns
Output 0 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

from hub to spoke

ping -f -l 1272 10.44.206.199

Ping wird ausgeführt für 10.44.206.199 mit 1272 Bytes Daten:
Antwort von 10.44.206.199: Bytes=1272 Zeit=11ms TTL=60

ping -f -l 1273 10.44.206.199

Ping wird ausgeführt für 10.44.206.199 mit 1273 Bytes Daten:
Paket müsste fragmentiert werden, DF-Flag ist jedoch gesetzt.

MHM Cisco World · ‎03-17-2022

I think the issue in MTU since above 1272 the packet need fragment!!!
check this issue.