MPLS link fast ping, no throughput

[silverstr8p]

I have a ISR 4321 at main office and one at remote office running MPLS. I just got it set up and remote office pings across link are around 3msec, but I can't get (almost) any throughput besides pings. Here's a network diagram of what I'm trying to do, I'm wondering if it has something to do with MSS/MTU? I tried adjusting the MSS, but I didn't really know what I was doing.

At the local office I'm running:

license boot level appxk9
license boot level securityk9
!
spanning-tree extend system-id
!
redundancy
 mode none
!         
vlan internal allocation policy ascending
!
pseudowire-class local
 encapsulation mpls
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface Tunnel1
 ip address 192.168.1.1 255.255.255.252
 ip tcp adjust-mss 1436
 mpls ip
 tunnel source GigabitEthernet0/0/0
 tunnel destination 5.6.7.8
!
interface GigabitEthernet0/0/0
 description WAN
 mtu 9216
 ip address 1.2.3.4 255.255.255.0
 negotiation auto
!
interface GigabitEthernet0/0/1
 no ip address
 negotiation auto
!
interface GigabitEthernet0/0/1.98
 encapsulation dot1Q 98
 xconnect 2.2.2.2 98 encapsulation mpls
!
interface GigabitEthernet0/0/1.200
 encapsulation dot1Q 200
 xconnect 2.2.2.2 200 encapsulation mpls
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 no ip address
 negotiation auto
!
!
interface Vlan1
 no ip address
 shutdown
!
ip forward-protocol nd
no ip http server
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 1.2.3.1
ip route 2.2.2.2 255.255.255.255 192.168.1.2
ip route 5.6.7.8 255.255.255.255 1.2.3.1
!
!
!
mpls ldp router-id Loopback0 force
!
!
...

 At the remote office I'm doing:

!
license boot level appxk9
license boot level securityk9
!
!
vlan internal allocation policy ascending
!
pseudowire-class vpn
 encapsulation mpls
!
pseudowire-class local
 encapsulation mpls
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface Tunnel1
 ip address 192.168.1.2 255.255.255.252
 ip tcp adjust-mss 1436
 mpls ip
 tunnel source GigabitEthernet0/0/0
 tunnel destination 1.2.3.4
!
interface GigabitEthernet0/0/0
 description WAN
 mtu 9216
 ip address 5.6.7.8 255.255.255.248
 negotiation auto
!
interface GigabitEthernet0/0/1
 description LAN
 no ip address
 negotiation auto
!
interface GigabitEthernet0/0/1.98
 encapsulation dot1Q 98
 xconnect 1.1.1.1 98 encapsulation mpls
!
interface GigabitEthernet0/0/1.200
 encapsulation dot1Q 200
 xconnect 1.1.1.1 200 encapsulation mpls
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 no ip address
 negotiation auto
!
interface Vlan1
 no ip address
 shutdown
!         
!         
ip forward-protocol nd
no ip http server
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 1.2.3.1
ip route 0.0.0.0 0.0.0.0 5.6.7.1
ip route 1.1.1.1 255.255.255.255 192.168.1.1
ip route 1.2.3.4 255.255.255.255 GigabitEthernet0/0/0
ip route 1.2.3.43 255.255.255.255 5.6.7.8
!
mpls ldp router-id Loopback0 force
!
...

How do I know if I'm getting TCP fragmentation? Would GRE help here?

[Georg Pauwen]

Hello,

at first glance, the MTU size of 9216 seems odd. What if you remove that (and use the default of 1500), or use the 'mpls mtu 1508' interface command, with 1508 being the recommended value for MPLS ?

[Giuseppe Larosa]

Hello @silverstr8p,

what kind of MPLS service have you bought from the service provider ?

Is it an ;MPLS L3 VPN service ?

If it is just use the standard MTU 1500 bytes on WAN interfaces of your routers and you should get better results as already noted.

You have a GRE tunnel with MPLS enabled on it directed to the remote office , what is the purpose of this tunnel ?

It would make sense if you had Internet handoff and you wanted to make a GRE over IPSEC LAN to LAN VPN.

Hope to help

Giuseppe

[silverstr8p]

I'm trying to just traverse public internet between two sites, so I haven't purchased an MPLS service from this provider, I'm just trying to figure out the best way to connect the main office with a remote office, even if it's got an internet connection from a different provider. I'm hoping the performance is far better than a OpenVPN connection or some such.

[Giuseppe Larosa]

Hello @silverstr8p,

as noted by Harold Ritter you are trying to create two EoMPLS pseudowires and then to carry them over a p2p GRE tunnel over the Internet.

Given the SW based platforms and the standard Internet Access you may achieve some better results with an IPSEC site to site VPN that is a routed solution, it requires different IP subnets on the two sites.

Otherwise you need to reduce the MTU on  all the involved PCs on both sites to avoid to create a too big packet as again recommended by Harold.

In any case your current routers can handle 100 - 200 Mbps of traffic not more with IPSec. For sure they do not scale to 10 GE speed for this kind of speed you would need two ASR 1000 that have hardware assisted encryption decrytion ESP module with appropriate license.

Please note that sending the frames over pseudowire over GRE on the public internet may give you some performance gain but your traffic will be clear text and subject to man in the middle . This is the reason why all routed or "bridged" solutions going over public internet use IPSec to protect the traffic.

Hope to help

Giuseppe

[paul driver]

Hello

---

@silverstr8p wrote:

How do I know if I'm getting TCP fragmentation? Would GRE help here?

---

You could initiate an extended ping with the Dont Fragment bit set, So as/when the max specified value is reached, fragmentation will incur:

Example - initiating an extended ping with a start value of 1450 with increments of 5  to a max mtu vlaue of 1490.

Ping
Protocol [ip]:
Target IP address: 10.1.12.2
Repeat count [5]: 2 <-------------------------------------
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 10.1.12.1
Type of service [0]:
Set DF bit in IP header? [no]: y
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]: y
Sweep min size [36]: 1450 <------------------------------------start of range
Sweep max size [18024]: 1490<----------------------------------end of range
Sweep interval [1]: 5 <----------------------------------------increment of each ping
Type escape sequence to abort.
Sending 18, [1450..1490]-byte ICMP Echos to 10.1.12.2, timeout is 2 seconds: <-----------------------Sweep count =See below
Packet sent with a source address of 10.1.12.1
Packet sent with the DF bit set
!!!!!!!..!!!!!!!..
Success rate is 77 percent (14/18), round-trip min/avg/max = 12/19/28 ms

[Harold Ritter]

A couple of issues here:

- The incoming L2 frame will potentially have a size larger than the tunnel interface MTU.

- Changing the MTU to something larger will not help either as you need to traverse the Internet, which normally has a path MTU of 1500. - The "ip tcp adjust-mss" will not help, as it is a L2VPN frame that you are trying to push through the tunnel interface, not a TCP segment.

- I do not think the ISR4321 supports fragmenting L2VPN traffic.

Normally service providers offering L2VPN services will have a larger path MTU through their core network to allow something like this, but as I mentioned above you can't change the Internet path MTU.

One option would be to reduce the MTU size (~1450-1454) on the workstations themselves. It is not elegant but it will work and given the current design there is not much more you can do.

Regards, 

[silverstr8p]

If I can't get it to work otherwise, I wonder if I could build a transparent proxy that alters MTU size as it traverses, though I wonder if there's a better way to do what I'm hoping to accomplish?

[Harold Ritter]

You can't change the size of the packets the workstations send to the ISR4321, unless you change the MTU on the workstation themselves. Why do you need to bridge between the two local segments. If you just need private connectivity between the two sites connected to Internet, why don't you just use IPsec between the two sites. The design you shared with us is rather complicated and will certainly bring its share of issues.

Regards,

[silverstr8p]

Will IPSec have decent performance? I know it's hard to get exact numbers, but I would like to move "significant" traffic across the link with low latency. Both ends have dedicated gigabit fiber but will eventually be 10gig. I want to transport traffic from 2 VLAN's, one for management subnet and one for raw traffic, would IPsec be a better way?