Cisco 3650 packet loss with Trunk port but not Access port

[Dan H]

Hello!

Not sure if anyone have experienced similar issue. We have 10 stacks of Cisco WS-C3650-24TD switches throughout our office, each stack has 2 x 3650 switches. The uplink of the 3650 stack connects to a pair of Cisco 4500X (running VSS) with 2 x 10Gbps LACP. The downlinks of the 3650 stack connects to multiple Aerohive SRS2148P switches with 4 x 1Gbps EtherChannel, same configuration applies to all 10 stacks of Cisco 3650 switches, basically the Cisco 3650 stack switches are running as Distribution layer switches, Cisco 4500X is the core layer and Aerohive switches are the Access layer.

Both uplink and downlinks of the Cisco 3650 switches are running trunk over the EtherChannel, so the Cisco 3650 is being used as a layer 2 switch, no routing at all, basically switching network traffic between the 4500X Core and Aerohive Access switches over multiple VLANs.

Network topology:

(Cisco 4500X VSS)--2x10Gbps--(Cisco 3650 stack)--4x1Gbps--(Aerohive switches)

This year we discovered very high number of output packet loss on the Cisco 3650, on all the interfaces connecting to the Aerohive switches, no packet loss on the 10Gbps interfaces towards the Cisco 4500X. Initially we thought maybe its the overkilled QoS design (1P+7Q), therefore we increase the buffer to 1200 which didn't help, we then removed QoS from all the interfaces, however this didn't resolve the issue. Long story short, in the end we discovered the root cause is when the interface connecting to Aerohive switch is configured with trunk.

In our test, we configured a LAB Aerohive switch. we removed EtherChannel configuration between the Cisco 3650 and the Aerohive switch, testing with only 1 single 1Gbps RJ45 cable. The results are:

0% packet loss if the link is configured with standard Access port.

interface GigabitEthernet1/0/23
switchport access vlan 7
switchport mode access
spanning-tree portfast
spanning-tree bpdufilter enable

Maximum of 0.28% packet loss if the link is configured with Trunk port and network traffic (VLAN 7) is running over the tagged VLAN.

interface GigabitEthernet1/0/24
switchport trunk native vlan 3
switchport mode trunk

Approx. 0.02% - 0.03% packet loss if the link is configured with Trunk port and the network traffic (VLAN 7) is running over the untagged VLAN (native).

interface GigabitEthernet1/0/24
switchport trunk native vlan 7
switchport mode trunk
end

However, if we replace the 1Gbps link with 10Gbps link between Cisco 3650 and Aerohive, then we have 0% packet loss!!

Configuration of the 2x10Gbps uplink to the Cisco 4500X on the Cisco 3650:

interface Port-channel111
description uplink to 4500X
switchport trunk allowed vlan 1,3,7,66,68,69,116,703,707,714,991,996
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk

Above tests were done using 2 x HP servers with 1Gbps NICs running Ubuntu server OS, using iperf3 with UDP packets and packet size 1400 and lower. 1 HP server running as iperf server connects to the Aerohive LAB switch (it is the only device on the Aerohive switch), 1 HP server running as iperf client connects to another stack of Cisco 3850 (not mentioned above) behind the Cisco 4500X core switch. We are confident no issue on the Cisco 4500X and Cisco 3850 stack, the root cause is between the Cisco 3650 and Aerohive, especially the Cisco 3650 output to Aerohive.

Topology during the test:

(HP server iperf3 client)--1Gbps--(Cisco 3850 stack)--3x10Gbps--(Cisco 4500X VSS)--2x10Gbps--(Cisco 3650 stack)--1Gbps--(Aerohive LAB switch)--1Gbps--(HP server iperf3 server)

Lastly, we have also tried upgrading the firmware and all the following version of firmware o the Cisco 3650 doesn't resolve the issue:

03.06.06E

16.03.07

16.09.08

16.12.08

Does anyone have experienced same/similar issue? Or any suggestions? Please help!

Thank you in advance.

[Joseph W. Doherty · Accepted solution]

I've used iperf a little bit but for most of my bandwidth testing, but I often found PCATTCP, using UDP packets, sufficient for my needs.  I.e. I'm not an iperf expert.

Anyway, back to the your observed results, your iperf source is generating a gig's worth of traffic while attached to a gig access port?  Or, you'll telling iperf to generate a gig's worth of traffic on a gig or 10g access port?  And if correct, this traffic, when it goes to/across a gig trunk port loses a small percentage of packets?  Also, you're using a MTU of 1400?

If so, "But it's hard to believe Cisco 3650 can't handle additional standard trunk port header and trunk traffic." that, yes, it's hard to believe because it's unlikely.  Again, realize an access port has more "payload" bandwidth than a trunk port (no trunk overhead protocols, no VLAN tagging overhead).  Tagging VLAN overhead, as a percentage, increases as frame size decreases.  (I.e. you might try same tests with minimum size packets and see if drop rates increases (this would create tagged frames with the highest percentage of overhead).  And/or try 1500 byte frame sizes and see if drop rates decrease a [very] small amount.

Hmm, a .1Q tag adds 4 bytes, so if your IP MTU is 1400, standard frame would be 1418 and tagged frame would be 1422, a difference of 0.282% - percentage look familiar?

If you're in the situation of sending more "payload" than a trunk link physically has bandwidth for, eventually, packets will be dropped.  Size of buffers only determine how long before drops occur, and/or perhaps avoid them for short duration bandwidth oversubscription bursts.

Miercom has a test report for the 3650/3850: https://miercom.com/pdf/reports/161201G.pdf, but this doesn't address port performance.

Miercom also has another test report for just the 3850: https://miercom.com/pdf/reports/20150225.pdf, which does address port performance.  Unknown whether same would apply to 3650, but good chance it would (as I suspect a 3650 is a reduced 3850).

[Dan H]

Thanks for your sharing, I think you are right, additional headers or packets required for trunk port comparing to access port. But it's hard to believe Cisco 3650 can't handle additional standard trunk port header and trunk traffic.