Cisco Switch 2950/2960 Latency Issue

[Hamidsattarrana]

Hi All

We have 2950 and 2960 switches installed in LAN network.

I want to know how can I monitor those switches health etc.

Issue is that the ping RTT should be 1 ms in LAN network but sometimes the latency got high like 1500ms or 1200ms. 

How can I find the issue? 

There are almost 8 Access switches 2950 and 2960 and they are connected to Cisco 3750-G Core switch.

Need suggestion. 

[Hamidsattarrana]

Dear Balaji Bandi,

where is the NMS nagios connected ? same switch ? i see these NMS some time alerts due to responce, we need to look what was the load that time and is there any routing change for the NMS to reach that IP.

Nagios is a Virtual Machines in ESXI Server which is directly connected with Core-Switch Cisco 3750-G.

Nagios and switches are on the same LAN subnet.

I have attached the diagram. Kindly check it. I think the issue is with 403-SW4. Kindly do check the Ethernet port speed. I have mentioned that about the fast Ethernet and Gigabit Ethernet Link.

[balaji.bandi]

Thank you for the infor,mation, that is very helpfull. As Long as you do not have any issue CORE Switch IP, 403-Sw2 IP, then i suspect below Link speed - not same  between   403-SW2 gigport - 403-Sw4 has Fast Ethernet

can able to post the configuration of that interface or setup up the speed to match each side static speed for 100MB.

is this latency show all time, only any specific time ?

[Hamidsattarrana]

HI!

Here are the CDP neighbors of Switch 403-SW2:

Device ID                                    Local Intrfce       Holdtme Capability        Platform                   Port ID
403-AP.407-Access Point           Gig 0/2              149           T I           AIR-AP114                    Gig 0
403-SW4                                   Gig 0/5              135           S I           WS-C2950S                  Fas 0/1
403-SW3                                   Gig 0/3              135           S I           WS-C2950S                  Fas 0/1
Core-SW.switch.com                  Gig 0/1              126           S I           WS-C3750G                 Gig 1/0/8

As you can see from the above output. Gig 0/5 & Gig 0/3 both are connected to switches with Fast Ethernet Links. And the output of show interface commands show "Output Drops" only for Gig 0/5 and 0/4 but not for Gig 0/1 & 0/2.  I guess that the issue is speed. Even both side are working on 100Mbps but might be possible that link is choking etc.

Output of Gig 0/5 and 0/3:

403-SW2#show int gig 0/5
GigabitEthernet0/5 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is d0c2.821b.c305 (bia d0c2.821b.c305)
Description: 403-SW4
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:05, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3561
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 8000 bits/sec, 1 packets/sec
5 minute output rate 41000 bits/sec, 27 packets/sec
2934453 packets input, 739866279 bytes, 0 no buffer
Received 106131 broadcasts (25440 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 25440 multicast, 0 pause input
0 input packets with dribble condition detected
9245087 packets output, 2653340325 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

403-SW2#show int gig 0/3
GigabitEthernet0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is d0c2.821b.c303 (bia d0c2.821b.c303)
Description: 403-SW3
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 3/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 795611
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 56000 bits/sec, 54 packets/sec
5 minute output rate 1535000 bits/sec, 228 packets/sec
55458935 packets input, 15024414273 bytes, 0 no buffer
Received 483761 broadcasts (260599 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 260599 multicast, 0 pause input
0 input packets with dribble condition detected
130781211 packets output, 89993594188 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

403-SW2#show int gig 0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is d0c2.821b.c301 (bia d0c2.821b.c301)
Description: Uplink-Core
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 ( 0 drops because both side are working on 1000Mbps)
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3659000 bits/sec, 524 packets/sec
5 minute output rate 543000 bits/sec, 256 packets/sec
218297765 packets input, 178015916777 bytes, 0 no buffer
Received 4295247 broadcasts (1757518 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1757518 multicast, 0 pause input
0 input packets with dribble condition detected
114201094 packets output, 36188992368 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

403-SW2#show int gig 0/2
GigabitEthernet0/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is d0c2.821b.c302 (bia d0c2.821b.c302)
Description: Access-Point
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:36, output 00:00:14, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 (Same no Drop)
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 440000 bits/sec, 175 packets/sec
5 minute output rate 1837000 bits/sec, 269 packets/sec
56541521 packets input, 20478422761 bytes, 0 no buffer
Received 1208609 broadcasts (644161 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 644161 multicast, 0 pause input
0 input packets with dribble condition detected
89658180 packets output, 86151615889 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Configurations of GIg 0/5 & 0/3:

403-SW2#show run int gig 0/5
Building configuration...

Current configuration : 212 bytes
!
interface GigabitEthernet0/5
description 403-SW4
switchport trunk allowed vlan 300,400
switchport mode trunk
logging event trunk-status
logging event spanning-tree
logging event status
no keepalive
end

interface GigabitEthernet0/3
description 403-SW3
switchport trunk allowed vlan 300,400
switchport mode trunk
logging event trunk-status
logging event spanning-tree
logging event status

no keepalive
end

interface GigabitEthernet0/2
description Access-Point
switchport access vlan 300
switchport mode access
logging event trunk-status
logging event spanning-tree
logging event status
no keepalive
end

interface GigabitEthernet0/1
description Uplink-Core
switchport trunk allowed vlan 300,400
switchport mode trunk
logging event trunk-status
logging event spanning-tree
logging event status
no keepalive
ip dhcp snooping trust
end

Is this latency show all time, only any specific time?

It shows mostly at peak hours.

[Joseph W. Doherty]

"I guess that the issue is speed. Even both side are working on 100Mbps but might be possible that link is choking etc."

If even a single gig port is feeding to a FE port, it's not uncommon for traffic to queue on the FE port.  If there's enough traffic, queues will overflow and drop the overflow traffic.

While traffic is queuing, latency will jump (as it waits to dequeue).  How much latency jumps will depend on how long a particular frame/packet waits in the queue.

1200 to 1500 ms would imply a long wait in a FE queue.

@100 Mbps, for 1500 ms, that's 10.5 million bits or 1,312,500 bytes or (about) 900 1,500 byte packets.  That number of packets seems rather large for a Catalyst 2K or 3K switch's port.

BTW, besides high ping latency, are there lost/dropped ping responses too?

(Also BTW, in a small LAN, ping response should often be under 1 ms, but many ping apps don't show smaller values than 1 ms, some do though.)

[balaji.bandi]

May be late reply due to clearing other stuff, if this is only peak hours - then i will ignore - if you are keen to really monitor ping is must for you to work, you can priotise the ping traffic, but personally i do not go that route.

You can use SNMP based monitoring rather ping, so light weight to monitor and give you status report of the device.

So you know the peak hours so this can added in your known list also if you like you can setup a EEM Script one of the next connect switch to ping and send email if this really gone down.

regards to grops, you need to drop down to 100MB see if that clears drops. after changing clear the counters and monitor.

[Joseph W. Doherty]

What's being pinged?

BTW, you realize ping is really a "are-you-alive" tool rather than a true network performance monitoring tool?  For this reason, hosts often give low-priority to replying to pings which can "skew" (as in increase) ping latency.

Also BTW, this is why Cisco developed SLA responders which provide a much, much more accurate end-to-end latency measurement.

[Hamidsattarrana]

Hi jeseph,

             Where should I configure SLA, Switches don't support SLA. I guess I should configure SLA on Core-Router? 

Need suggestion.

[Joseph W. Doherty]

You might configure SLA on any supported device which can provide you SLA stats that would be of interest to you.

[ohassairi]

why do you think the delay is from the network

in many cases the delay is from the server.