Very Slow Upload Speeds on Cisco 3850

[NetworkNewbie]

Hello,

I work at a University and for about a week now, I have noticed that the upload speeds at one of our resident halls is extremely low. So low, that I just ran a speed test and received .80Mbps. Usually we receive about 700Mbps upload, so there is definitely something going on. It has me stumped. Our setup is something like this.. On the first floor, we have a Cisco 9300 that gets a fiber connection from our core switch at our main building and from there, we have a 10G fiber connection running to each of the floors. I have checked both interfaces for output errors, collisions, CRC, tx and rx load and everything seems normal. I have also tried connecting to different vlans, and still get the same result.

One thing to note.. if I run a speed test from the first floor, the speeds are perfect, so it makes me think there is an issue with the connections from the first floor to the others. I have double checked speed and duplex settings. The switch shows auto full-duplex and auto 1000 and my PC also shows 1Gb when I view the ethernet stats. I will admit, I am still a newbie, so any insight is appreciated. If you need more information, ask away. What else would you guys check if you ran into an issue like this?

Thank you!

[NetworkNewbie]

Okay I just ran another test on the first floor switch and the upload speeds are slow too.. so it looks like the entire building is getting slow upload speeds

[Leo Laohoo]

Check the ports: 3650/3850 are notorious for QoS.
Check the ports for speed/duplex mismatch.
Check the uplinks for line errors.

[NetworkNewbie]

Hi Leo,

Thanks for replying. 

This interface is from our Core switch to the device in question:

TenGigabitEthernet5/9 is up, line protocol is up (connected)
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 250/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-LR
input flow-control is on, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 50w3d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4414000 bits/sec, 1020 packets/sec
5 minute output rate 16626000 bits/sec, 2475 packets/sec
101218256072 packets input, 99805682385583 bytes, 0 no buffer
Received 330199162 broadcasts (274487278 multicasts)
0 runts, 0 giants, 0 throttles
79186507 input errors, 79186507 CRC, 4680650 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
106229168343 packets output, 82915104026797 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

This interface is from the device in question back to the core switch:

TenGigabitEthernet1/0/1 is up, line protocol is up (connected)
Description: uplink
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-LR
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 12w1d
Input queue: 2/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 16612000 bits/sec, 2481 packets/sec
5 minute output rate 4411000 bits/sec, 1099 packets/sec
21063040585 packets input, 15063324311919 bytes, 0 no buffer
Received 1856478861 broadcasts (1577067590 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1577067590 multicast, 0 pause input
0 input packets with dribble condition detected
21288741754 packets output, 21477293059895 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

From what I can see, I don't see any errors, but you might see something different. As for QoS, how would I check for that?

Thank you!

[Leo Laohoo · Accepted solution]

---

@NetworkNewbie wrote:

reliability 250/255

---

I can see line errors just fine.  Ten 5/9 has a layer 1 issue. 

---

@NetworkNewbie wrote:

79186507 input errors, 79186507 CRC, 4680650 frame, 0 overrun, 0 ignored

---

Look at the input errors.  

[NetworkNewbie]

My focus has been looking for output errors and not input errors. So if I replace that cable, those issues should go away?

[Leo Laohoo · Accepted solution]

---

@NetworkNewbie wrote:

So if I replace that cable, those issues should go away?

---

My focus is to find and fix issues as they arise.  

I'm saying now, there is a layer 1 issue.  Cable isn't the only thing that will cause line errors. 

[NetworkNewbie]

You are correct.

We swapped the cables from the core switch because that's where we saw the errors, but the input errors were still getting higher. So we looked at the other end and replaced that cable and after clearing the counters, we haven't seen any input errors yes and the speeds are back to normal.

Thanks again!

[Leo Laohoo]

Happy to hear the issue has been fixed.
Thanks for feedback.