WAN Slow speeds in one direction

[Daz_McD]

Hi 

Hi Guys - Long time reader, 1st time poster here....

I have an issue with our MPLS WAN, that is driving me crazy. 

Site A has 2 x Cat 4500x's primary/secondary in VRRP, going to a 300/1gb (primary) 100/1gb(secondary).

Site B has 2 x Dell N3048P primary/secondary in VRRP, going to a 300/1gb (primary) 100/1gb(secondary). 

In between is BGP handled by our ISP.

Sending a 1gb file from Site B to Site A it takes sub 30 secs.

Send it from Site A to Site B takes between 4-6mins.

• Changed all the cables - no change.
• Swapped all SFP's - no change.
• Tried hardcoding the speed/duplex - no change.
• We have tried hardcoding the MTU (1500, 1522, 1548, 9000) - slight change, the time became a constant 4mins but this is  obviously still not good enough.
• We've gone direct from the ISP's router on one site to ISP router on the other site and that is coming in at sub 20secs. So this seems to rule out the ISP having the issue.
• We've turned of the "TCP Autotuning" - no change.

So I'm opening it up to the community to see if anyone has any other options I've not explored? Any help would be really appreciated.

[Georg Pauwen]

Hello,

can you post the output of a traceroute from Site A to Site B, and vice versa ? Traffic might be taking a different route each way...

[Daz_McD]

Here you go:

Site A - Site B

LYT_Core_Sw01#traceroute 10.10.20.51
Type escape sequence to abort.
Tracing the route to 10.10.20.51
VRF info: (vrf in name/id, vrf out name/id)
1 10.10.10.6 0 msec 4 msec 4 msec
2 100.64.100.225 12 msec 8 msec 12 msec
3 100.64.73.89 8 msec 12 msec 12 msec
4 100.64.73.90 16 msec 20 msec 16 msec
5 10.10.20.51 16 msec 20 msec 16 msec
LYT_Core_Sw01#

Site B to Site A

Lpool_Core_Sw01#traceroute 10.10.10.1

Traceroute to 10.10.10.1 ,30 hops max 0 byte packets:

1 10.10.20.56 <1 ms <1 ms <1 ms
2 100.64.73.89 <1 ms <1 ms <1 ms
3 100.64.100.225 <1 ms <1 ms <1 ms
4 100.64.100.226 <1 ms <1 ms <1 ms
5 10.10.10.1 <1 ms * <1 ms

Hop Count = 5 Last TTL = 5 Test attempt = 15 Test Success = 14

Lpool_Core_Sw01#

As you can see they seem to take the same route.

[Georg Pauwen]

Hello,

traceroute looks the same both ways indeed. Is the 4500 your ISP facing device ?

Can you post the output of 'show interfaces x' where 'x' are the relevant interfaces on the 4500s ?

[Daz_McD]

Yes the 4500's are the ISP facing devices:

Primary:

LYT_Core_Sw01#
LYT_Core_Sw01#sh int t2/7
TenGigabitEthernet2/7 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 881d.fc5e.df6e (bia 881d.fc5e.df6e)
Description: Zen_Primary_WAN_Link
MTU 1522 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 4/255, rxload 5/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseT
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:11, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 21461000 bits/sec, 3492 packets/sec
5 minute output rate 16038000 bits/sec, 2957 packets/sec
5168153368 packets input, 3150640832125 bytes, 0 no buffer
Received 3055559 broadcasts (3055004 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
7381914108 packets output, 7403027120463 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
LYT_Core_Sw01#
LYT_Core_Sw01#

Secondary:

LYT_Core_Sw02#
LYT_Core_Sw02#sh int t2/7
TenGigabitEthernet2/7 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 4c4e.358c.ac86 (bia 4c4e.358c.ac86)
Description: Zen_Secondary_WAN_Link
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseT
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:09, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 4018599
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 2 packets/sec
5 minute output rate 112000 bits/sec, 77 packets/sec
10231137 packets input, 2287949448 bytes, 0 no buffer
Received 4853888 broadcasts (4853639 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
353380469 packets output, 220518325540 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
LYT_Core_Sw02#
LYT_Core_Sw02#

[Georg Pauwen]

Hello,

this looks odd:

LYT_Core_Sw01#
LYT_Core_Sw01#sh int t2/7
TenGigabitEthernet2/7 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 881d.fc5e.df6e (bia 881d.fc5e.df6e)
Description: Zen_Primary_WAN_Link
MTU 1522 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

On your secondary switch, you haveoutput drops:

Total output drops: 4018599

Do you have any sort of QoS enabled ?

[Daz_McD]

No there is no QoS enabled.

With regards tot he MTU on the primary, ive tried it at various settings.

Not noticed the drops on the secondary, got to be honest I've been focused on the primary. But our ISP have confirmed that all traffic is passing via the primary.

[Joseph W. Doherty]

Lots of things can cause that, including issues with the hosts doing the transfers (i.e. might not be a "network" issue at all).

Packet capture analysis can often reveal what's happening.