Replacing Stacked 3750G switches

[BashedRoot]

Curently my setup is simple.

Cab 1 = primary 3750G switch

Cab 2 = stacked 3750G switch

If I get a quick replacement 3750 for the stacked switch in cab 6, would that one be a simple swap (just re-cabling) or require re-configuration? It’s stacked, keep in mind. I create, edit VLANs via one single login, the primary so my guess it would be a simple replacement swap?

Reason for my replacement is the 3750 in cab 2 (secondary stacked) seems to have major packet loss and latency. It seems the primary in cab 1 is all good across the board. Also, how do I check if there's some sort of flow control / QoS enabled that might be causing this?

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

A 3750 stack operates as if it were a single unit with the stack member "appearing" like a chassis line cards.  Swapping a single unit, if the replacement isn't configured, should allow the replacement to "pick up" the prior unit's config.  (This, though, may require the replacement to be the same model 3750 as the one being replaced.  If not, you would need to configure its ports after it joins the stack.)

As to packet loss and latency, if a egress port is dealing with traffic bursts, the 3750 series is a bit infamous for dropping packets, especially if default QoS is active, due to the 3750's small buffer capacity.  For transient bursts, sometimes tuning a 3750's config can dramatically decrease egress drops.

[BashedRoot]

Thanks for the info. How do I disable QoS, or check if it's even there?

I can’t determine which model is which switch though, and curious what the difference is between WS-C3750G-24T-S and WS-C3750G-24TS-S

Cisco3750#show inventory

NAME: "1", DESCR: "WS-C3750G-24T"

PID: WS-C3750G-24T-S   , VID: V05  , SN:

NAME: "2", DESCR: "WS-C3750G-24TS"

PID: WS-C3750G-24TS-S  , VID: G0   , SN:

NAME: "GigabitEthernet2/0/28", DESCR: "1000BaseSX SFP"

PID: Unspecified       , VID:      , SN:

Switch#  Role   Mac Address     Priority Version  State

----------------------------------------------------------

*1       Master 001f.6c8a.9380     15     0       Ready              

 2       Member 0012.01db.f600     14     0       Ready     

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

How to disable 3750 QoS?  In config mode, no mls qos.

How to check if 3750 QoS enabled?  Either show mls qos or look for "mls qos" in config.

WS-C3750G-24T-S is a 1 RU

WS-C3750G-24TS-S is a 1.5 RU with 4 SFP ports.

[BashedRoot]

Truly appreciate your help.

I'm not sure what the below means.

Cisco3750#show mls qos
QoS is disabled
QoS ip packet dscp rewrite is enabled

I see it says disabled, but what about the 2nd line, dscp rewrite?

[Joseph W. Doherty]

I believe that has nothing to due with your issue.

[BashedRoot]

Ok here's the strange thing.

1. Several Dell 610 servers are crawling when it comes to speed, simple things like wget or cachefly tests. They're set on GigE / Full Duplex on the switches. NIC drivers updated, rebooted, running Centos 6.8

2. Somehow, I found one particular server on 100Mbit speed, but consistently pushing nearly 40MBps every day. How is that even possible?

My Cisco processor is constantly at 99%. See attached. If I issue a reload, it shoots up rather quickly.

http://i.imgur.com/JLSHlrH.png

[Leo Laohoo]

Check the interface of each server.  See if there are any "Output Drops".

[BashedRoot]

Could you please explain to me explain how I would go about doing that?

[Leo Laohoo]

Could you please explain to me explain how I would go about doing that?

Perform the command "sh interface <BLAH>" and see if there are any "Total Output Drops".  

Total Output Drops are an interface-based counters where the interface could no longer "hold on" to the data and drops the packets.  In an ideal environment, Total Output Drops should be a ZERO. 

Another thing, Cisco Catalyst range of switches were not designed for server duties.  They are notorious for very shallow buffers and start dropping packets when high-speed traffic starts pummeling them.

[BashedRoot]

For some reason I don't see any line that says "Total Output Drops"

Example:

Cisco3750#sh interface Gi1/0/7
GigabitEthernet1/0/7 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001f.6c8a.9387 (bia 001f.6c8a.9387)
  Description: Server 111
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 10000 bits/sec, 9 packets/sec
  5 minute output rate 8000 bits/sec, 10 packets/sec
     5091180 packets input, 1915463255 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     5243439 packets output, 694967769 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

GigabitEthernet1/0/7 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001f.6c8a.9387 (bia 001f.6c8a.9387)
  Description: Server 111
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 10000 bits/sec, 9 packets/sec
  5 minute output rate 8000 bits/sec, 10 packets/sec
     5091180 packets input, 1915463255 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     5243439 packets output, 694967769 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

In RED.

[BashedRoot]

Thanks. Just ran this on another box, the one that claims is having latency issues.

Cisco3750#sh interface Gi1/0/21
GigabitEthernet1/0/21 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001f.6c8a. (bia 001f.6c8a.9395)
  Description: Server 11
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 23:36:47, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 44000 bits/sec, 86 packets/sec
     57706331 packets input, 32444097669 bytes, 0 no buffer
     Received 43355 broadcasts (43325 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 43325 multicast, 0 pause input
     0 input packets with dribble condition detected
     67902039 packets output, 34141503953 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

  Last input 23:36:47, output 00:00:00, output hang never

Not a latency issue.  Server issue. 

No response from the server for almost 24 hours????  Only a server issue can do this.

Total Output Drops is ZERO.  Look at the significant difference between the Packets Input vs Packets Output.  `tis a server issue.

[Joseph W. Doherty]

Could you post switch's CPU stats when it's running at 99%; both normal stats sorted (top 5 processes should be enough) and CPU history.

3750 CPU utilization shouldn't be that high.  Wondering whether you're doing software switching.