Cisco SG300 series poor performance when connected to Catalyst 3650

[CyberNBD]

Hello,

I have a network consisting of several Catalyst WS-C3650-48-FQ-L switches to which we sometimes want to add small drop-switches for a few devices.  Since we have some spare SG300-10MPP switches I thought to use these but somehow these SG300 switches seem to cause massive performance issues for the devices connected to them.

On their own the SG300 switches work fine.  Connecting all devices to the 3650's also poses no issues.  I did all troubleshooting I could think of including creating a small stand-alone network with some spare switches in default config, but I get the exact same behaviour. There are no interface errors.  From default config tried disabling EEE, POE etc. on the inter-switch ports.  Set them to default 1G, checked STP etc. but the problems persists. 

Only thing I am noticing is unknown protocol dropped packets on the 3650 port to which the SG is connected.  Could this be a cause of the issues, and where should I look further to resolve this?  Anything else I am missing what could cause the incompatibility between both series of switches?

[pieterh]

the SG300 supports RSTP an MST (default RSTP)
the 3650 supports (rapid) PVST+ and MST  (default PVST+)

-> best to migrate to a common STP mode when using multiple vlans
or using an access-port (with bpduguard) on the 3650 where a SG300 is connected to pass only a single vlan

but there should be entries in the  console logging that refers to this  if there is a mismatch?

[CyberNBD]

no errors in console logging but I checked STP settings/status to be sure:

3650 is indeed in rapid-pvst mode and SG in RSTP.  I changed both to MST to be sure but nothing really changes performance-wise.  I already changed from trunk to a single vlan on access ports earlier in my troubleshooting.

Port 23 on the 3650 and port 8 on the SG is the inter-switch link

SW-3650#sh span

MST0
  Spanning tree enabled protocol mstp
  Root ID    Priority    32768
             Address     002c.c8bb.bc00
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     002c.c8bb.bc00
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/13            Desg FWD 200000    128.13   P2p Edge
Gi1/0/15            Desg FWD 2000000   128.15   P2p Edge
Gi1/0/17            Desg FWD 200000    128.17   P2p Edge
Gi1/0/19            Desg FWD 20000     128.19   P2p Edge
Gi1/0/23            Desg FWD 20000     128.23   P2p
Gi1/0/43            Desg FWD 20000     128.43   P2p Edge
Te1/1/1             Desg FWD 2000      128.49   P2p Edge
Po10                Desg FWD 1000      128.2386 P2p

SW-SG#sh span

Spanning tree enabled mode MSTP
Default port cost method:  long
Loopback guard:   Disabled

Gathering information ..........
###### MST 0 Vlans Mapped: 1,4-5,11-12,21-22,31,41,91-92
CST Root ID    Priority    32768
               Address     00:2c:c8:bb:bc:00
               Path Cost   20000
               Root Port   gi8
               This switch is the IST master
               Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec
Bridge ID      Priority    61440
               Address     18:8b:45:2e:d7:e6
               Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec
               Max hops    20
  Name     State   Prio.Nbr    Cost    Sts   Role PortFast       Type
--------- -------- --------- -------- ------ ---- -------- -----------------
gi1       enabled  128.49    20000    Frw    Desg Yes      P2P Intr
gi2       enabled  128.50    2000000  Dsbl   Dsbl No       -
gi3       enabled  128.51    2000000  Dsbl   Dsbl No       -
gi4       enabled  128.52    2000000  Dsbl   Dsbl No       -
gi5       enabled  128.53    20000    Frw    Desg Yes      P2P Intr
gi6       enabled  128.54    2000000  Dsbl   Dsbl No       -
gi7       enabled  128.55    2000000  Dsbl   Dsbl No       -
gi8       enabled  128.56    20000    Frw    Root No       P2P Bound (RSTP)

[pieterh]

gi8       enabled  128.56    20000    Frw    Root No       P2P Bound (RSTP)

RSTP is unexpected here ?

MST 0 Vlans Mapped: 1,4-5,11-12,21-22,31,41,91-92

MST config needs to be equal between same instances

[CyberNBD]

---

@pieterh wrote:

gi8       enabled  128.56    20000    Frw    Root No       P2P Bound (RSTP)

RSTP is unexpected here ?

Agreed, but both switches are in MSTP mode, how's that possible?   Tried reloading everything but stays like this.

---

@pieterh wrote:

MST 0 Vlans Mapped: 1,4-5,11-12,21-22,31,41,91-92

MST config needs to be equal between same instances

---

Both switches have all these vlans.  I can do:

spanning-tree mst configuration
instance 0 vlan 1,4-5,11-12,21-22,31,41,91-92

 on the 3650 but it doesn't show up in config (perhaps because it's just all vlans?)

STP config on both switches is mostly default except for the priority on the SG300 to avoid it becoming root bridge

[pieterh]

you could move vlan 1,4-5,11-12,21-22,31,41,91-92 to MST instance 1 (on all switches)
unmentioned vlans will remain with the default instance 0
but let's keep the current config for the moment

speed 1000

if you set the speed fixed on an interface you need to do this on both sides of the link
that is :
     auto <-> auto
or
     fixed<-> fixed,
=> do not use fixed <-> auto
the other end will still try to autonegotiate periodically, this may be the source of your performance issue

setting speed fixed may also disable auto MDI-MDI-X, if so you need a cross-cable between two switches

[balaji.bandi]

how is your configuration

show run interface G1/0/23 and g8

what vlans allowed

as mentioned you need to match the vlan also.

[CyberNBD]

3650:

interface GigabitEthernet1/0/23
 switchport access vlan 31
 switchport mode access
 switchport nonegotiate
 speed 1000
end

SG300:

interface gigabitethernet8
 no eee enable
 flowcontrol on
 no eee lldp enable
 no green-ethernet energy-detect
 switchport mode access
 switchport access vlan 31

[balaji.bandi]

If both are gigabit you do not need speed command.

Cisco SG300 series poor performance   -

So the gateway is on 3650 - how is your Ping, do you see any interface errors ?

[CyberNBD]

Ping is fine, no interface errors at all.  All hosts are reachable and with little traffic there are no issues.  With higher bandwidths (let's say 100mbit+) we start to see issues like stuttering and dropped frames in realtime video on the hosts connected to the SG (both inbound and outbound traffic have issues).  Everything connected to one or more 3650's is fine. 

It's a really basic L2 setup, no routers, firewalls or gateways.  Just a few switches and vlans.  Even with no traffic on all other vlan's the issue remains.

[balaji.bandi]

Make sure you disable any QoS or any macros configured on the end port - since by default these switches are enabled macros.

[CyberNBD]

Macros have been disabled globally on the SG using macro auto disabled.  I now also disabled qos using the no qos command.  No change in behaviour unfortunately.

The 3650's seem to have all of this disabled by default

[balaji.bandi]

ok now we got in to tricky postion - we need more informaton now.

1. Low speed from where to where ?

2. how is your speed  device located in 3650 to device in small business switchs ?

post show interface gi 1/0/23

show interface gi8 output

[CyberNBD]

---

@balaji.bandi wrote:

1. Low speed from where to where ?

2. how is your speed device located in 3650 to device in small business switchs ?

Basically any device connected to the SG has performance issues to the main devices on the 3650.  Current test setup is two 3650 switches with each a video server connected to te1/1/1.  Both switches are interconnected using a port-channel on te1/1/2 and 3 on each switch.  Encoders/decoders/client devices connected to the 1g ports of the 3650's.  This all runs fine with no issues.

For reference about traffic and tested/approved on a setup using only 3650's:
Encoders/decoders generate/consume traffic anywhere between 50mbps and 300mbps.  Client devices depends on usage but testcase pushed to 800mbps bi-directional.  Videoservers tested to about 6gbps testcase.  Everything in a mix (8 encoder/decoder, 2 client devices, 2 video servers works fine with no issues at all.

Now when I add the SG300 dropswitch and move some devices from the 3650's to the SG this is where the issues start and these devices won't work properly anymore.  For example moving one encoder to the sg300 will work fine with low traffic but add a second and problems start.  I would say 100mbps is probably the max to push through the SG before having issues. The problems are bidirectional.  So traffic flowing from any video server on any 3650 to a decoder on SG does not work well and traffic from an encoder on the SG to any video server on any 3650 also has issues.  Connect these exact encoders directly to one of the 3650's and issues are gone.

I confirmed cable between SG and 3650 is fine and also tried swapping cables and ports.

---

@balaji.bandi wrote:

post show interface gi 1/0/23

---

SW-3650#sh int gi1/0/23
GigabitEthernet1/0/23 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 002c.c8bb.bc17 (bia 002c.c8bb.bc17)
  Description: NDI
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 39/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:08, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 24334337
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 739000 bits/sec, 1339 packets/sec
  5 minute output rate 153156000 bits/sec, 13158 packets/sec
     655214 packets input, 45007108 bytes, 0 no buffer
     Received 474 broadcasts (460 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 460 multicast, 0 pause input
     0 input packets with dribble condition detected
     6462242 packets output, 9399547377 bytes, 0 underruns
     Output 4362 broadcasts (0 multicasts)
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

---

@balaji.bandi wrote:

show interface gi8 output

---

No such command but this is probably what we're looking for?

SW-PTZ-003#sh int coun gi8

      Port       InUcastPkts  InMcastPkts  InBcastPkts    InOctets
---------------- ------------ ------------ ------------ ------------
      gi8          7428833        2581         4620     10818823219

      Port       OutUcastPkts OutMcastPkts OutBcastPkts  OutOctets
---------------- ------------ ------------ ------------ ------------
      gi8           752063        472           14        51692550

Alignment Errors: 0
FCS Errors: 0
Single Collision Frames: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Late Collisions: 0
Excessive Collisions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

I now see the output drops on gi1/0/23.  This probably is a clear sign of issues.  But what could be the cause?

[balaji.bandi]

here is some causing the issue : ( you identified already)

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 24334337

Do you have IGMP enabled for the Multicast ?

show run (from both the device help us).