Hello!
Not sure if anyone have experienced similar issue. We have 10 stacks of Cisco WS-C3650-24TD switches throughout our office, each stack has 2 x 3650 switches. The uplink of the 3650 stack connects to a pair of Cisco 4500X (running VSS) with 2 x 10Gbps LACP. The downlinks of the 3650 stack connects to multiple Aerohive SRS2148P switches with 4 x 1Gbps EtherChannel, same configuration applies to all 10 stacks of Cisco 3650 switches, basically the Cisco 3650 stack switches are running as Distribution layer switches, Cisco 4500X is the core layer and Aerohive switches are the Access layer.
Both uplink and downlinks of the Cisco 3650 switches are running trunk over the EtherChannel, so the Cisco 3650 is being used as a layer 2 switch, no routing at all, basically switching network traffic between the 4500X Core and Aerohive Access switches over multiple VLANs.
Network topology:
(Cisco 4500X VSS)--2x10Gbps--(Cisco 3650 stack)--4x1Gbps--(Aerohive switches)
This year we discovered very high number of output packet loss on the Cisco 3650, on all the interfaces connecting to the Aerohive switches, no packet loss on the 10Gbps interfaces towards the Cisco 4500X. Initially we thought maybe its the overkilled QoS design (1P+7Q), therefore we increase the buffer to 1200 which didn't help, we then removed QoS from all the interfaces, however this didn't resolve the issue. Long story short, in the end we discovered the root cause is when the interface connecting to Aerohive switch is configured with trunk.
In our test, we configured a LAB Aerohive switch. we removed EtherChannel configuration between the Cisco 3650 and the Aerohive switch, testing with only 1 single 1Gbps RJ45 cable. The results are:
0% packet loss if the link is configured with standard Access port.
interface GigabitEthernet1/0/23
switchport access vlan 7
switchport mode access
spanning-tree portfast
spanning-tree bpdufilter enable
Maximum of 0.28% packet loss if the link is configured with Trunk port and network traffic (VLAN 7) is running over the tagged VLAN.
interface GigabitEthernet1/0/24
switchport trunk native vlan 3
switchport mode trunk
Approx. 0.02% - 0.03% packet loss if the link is configured with Trunk port and the network traffic (VLAN 7) is running over the untagged VLAN (native).
interface GigabitEthernet1/0/24
switchport trunk native vlan 7
switchport mode trunk
end
However, if we replace the 1Gbps link with 10Gbps link between Cisco 3650 and Aerohive, then we have 0% packet loss!!
Configuration of the 2x10Gbps uplink to the Cisco 4500X on the Cisco 3650:
interface Port-channel111
description uplink to 4500X
switchport trunk allowed vlan 1,3,7,66,68,69,116,703,707,714,991,996
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
Above tests were done using 2 x HP servers with 1Gbps NICs running Ubuntu server OS, using iperf3 with UDP packets and packet size 1400 and lower. 1 HP server running as iperf server connects to the Aerohive LAB switch (it is the only device on the Aerohive switch), 1 HP server running as iperf client connects to another stack of Cisco 3850 (not mentioned above) behind the Cisco 4500X core switch. We are confident no issue on the Cisco 4500X and Cisco 3850 stack, the root cause is between the Cisco 3650 and Aerohive, especially the Cisco 3650 output to Aerohive.
Topology during the test:
(HP server iperf3 client)--1Gbps--(Cisco 3850 stack)--3x10Gbps--(Cisco 4500X VSS)--2x10Gbps--(Cisco 3650 stack)--1Gbps--(Aerohive LAB switch)--1Gbps--(HP server iperf3 server)
Lastly, we have also tried upgrading the firmware and all the following version of firmware o the Cisco 3650 doesn't resolve the issue:
03.06.06E
16.03.07
16.09.08
16.12.08
Does anyone have experienced same/similar issue? Or any suggestions? Please help!
Thank you in advance.
