- Cisco Community
- Technology and Support
- Networking
- Switching
- Pinpoint Switch Bottleneck
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Pinpoint Switch Bottleneck
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2018 02:24 PM - edited 03-08-2019 04:55 PM
This question is largely academic in nature - I have an OLD Cisco 2950 switch that is passing an increasingly high level of traffic and I'm certain that it's bottle-necking my network, but how would I prove that? I'm replacing the switch with a new gigabit unit next week, but how could I have identified this performance issue via metrics?
The switch is 10/100, with a pair of gigabit ports, and all of the traffic that is flowing through it internet-type of traffic (5-40mbps, from various ISP's, firewalls, DMZ, etc.) The traffic is segregated through numerous VLAN's, but otherwise there is nothing extraordinary about the configs. In theory all of the traffic should be well within the specs of a 100Mbps switch.
However, there is DMZ traffic passing through this switch, which includes a lot of SQL queries (with very large result-sets) as well as some proprietary TCP connections that can return fair amounts of data.
We noticed that some of our traffic was getting retry errors, and we were not getting the full bandwidth capacity from our ISP's when traversing this switch. We reconfigured one of the paths to use the gigabit ports which helped that traffic (even though it's only a 40/40mbps circuit), and when I did that, I noticed that my console CLI slowed to a crawl - the 9600baud looked about like 1200baud.
I pulled some stats that I thought might be helpful, but I just don't find a smoking gun that PROVES that there's a problem with this switch. I've attached some snips that maybe will mean more to others than myself.
show int
GigabitEthernet0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 001f.279a.9f99 (bia 001f.279a.9f99)
Description: InetNVT5
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1067000 bits/sec, 124 packets/sec
5 minute output rate 156000 bits/sec, 93 packets/sec
65554738 packets input, 55780401735 bytes, 0 no buffer
Received 387 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
58780936 packets output, 36441903384 bytes, 0 underruns
0 output errors, 714906 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 563592 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 001f.279a.9f9a (bia 001f.279a.9f9a)
Description: InetNVT5
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 156000 bits/sec, 92 packets/sec
5 minute output rate 1067000 bits/sec, 125 packets/sec
58572674 packets input, 36779751576 bytes, 0 no buffer
Received 6 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
65790467 packets output, 55792984590 bytes, 0 underruns
0 output errors, 1926 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 3598 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
show int counter
Port InOctets InUcastPkts InMcastPkts InBcastPkts Fa0/1 6280250775933 9231296959 0 639 Fa0/2 1070635 6603 1554 5357 Fa0/3 6710244949295 9488311808 1049 4172 Fa0/4 77414 598 162 137 Fa0/5 6237948903196 16471810463 33 10276718 Fa0/6 19997607446810 21191741146 56301631 3346178839 Fa0/7 1069327834239 2043086938 18932982 59688798 Fa0/8 164963391712 999996896 109 2786 Fa0/9 35921 169 73 161 Fa0/10 30015663681418 46248128747 87 10604 Fa0/11 40135624559468 51405591747 38 2249161 Fa0/12 35084583062 209872036 48 203 Fa0/13 7537226578 48714042 715 3054 Fa0/14 1516601936 1688149 0 2 Fa0/15 8970207000875 16010043012 24056240 762486 Fa0/16 28012264871851 45653370694 123913396 117415909 Fa0/17 14790057083901 29638689004 103454467 10811758 Fa0/18 3171459106800 6169817338 9832759 18191764 Fa0/19 130042520174114 122350475789 3041860602 6005082 Fa0/20 5405543208944 13116181191 37604590 1475890 Fa0/21 77592569004 378822460 0 8179 Fa0/22 1114541769744 1295099614 7520073 245742 Fa0/23 1267360939321 2085553221 1989578 332521 Fa0/24 1890512734704 3901137214 39040055 949204 Gi0/1 55869324831 65641471 0 387 Gi0/2 36794759394 58638793 0 6 Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts Fa0/1 6711382043763 9488305658 17773818 8649 Fa0/2 869500 5516 5371 418 Fa0/3 6168937213911 9038719975 17771682 6122 Fa0/4 60522 34 893 13 Fa0/5 19731849145164 22088495995 74596357 3405867532 Fa0/6 5112038665722 15473760141 18926060 69961560 Fa0/7 1495555797961 1853090397 56298293 3356393936 Fa0/8 1395010160426 1291304330 35416404 439598513 Fa0/9 429799 131 21 6272 Fa0/10 39962102508917 51110968894 645 2252402 Fa0/11 29526835538711 46500257680 700 13841 Fa0/12 168201258610 268427142 61 955659 Fa0/13 5185626838 24719583 40 1060779 Fa0/14 432246260 1555913 730 2263015 Fa0/15 23816780973184 20756205026 2797702874 151503908 Fa0/16 53941853629018 52869111737 2697704913 34850530 Fa0/17 40513136195324 37670565471 2718203782 141455032 Fa0/18 8799986657308 7535337293 2811786674 134075422 Fa0/19 44537393134174 95331334270 347139380 150186526 Fa0/20 19208301030895 17832175464 657884877 4995227 Fa0/21 1841723422709 564665337 3388867008 156186829 Fa0/22 947878120482 1147558653 868714191 10560127 Fa0/23 2098800811261 2332341031 725084628 6769560 Fa0/24 4453444449215 4552655768 555145889 3140609 Gi0/1 36457052034 58075471 237226 6 Gi0/2 55882507436 65637965 237052 159
show int counter err
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Fa0/1 3 3996 0 9034 0 0 Fa0/2 0 0 0 0 0 0 Fa0/3 0 1 0 1 0 8608 Fa0/4 0 0 0 0 0 0 Fa0/5 0 8 0 8 0 0 Fa0/6 3 5 0 8 0 0 Fa0/7 0 0 0 3 3 0 Fa0/8 28 468 0 498 0 0 Fa0/9 0 0 0 1 0 0 Fa0/10 0 934203121 0 934203127 0 0 Fa0/11 0 0 0 0 0 0 Fa0/12 0 2197 0 2198 0 0 Fa0/13 0 1220 0 1220 0 0 Fa0/14 0 0 0 0 0 0 Fa0/15 0 1 0 1 0 2029748 Fa0/16 0 0 0 0 0 3481479 Fa0/17 0 0 0 2 0 103346 Fa0/18 0 0 0 4 0 29858 Fa0/19 0 0 0 0 0 2461307 Fa0/20 0 0 0 0 0 0 Fa0/21 0 1 0 14 0 0 Fa0/22 0 0 0 0 0 0 Fa0/23 0 1 0 1 0 0 Fa0/24 0 0 0 0 0 0 Gi0/1 0 0 0 0 0 0 Gi0/2 0 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Fa0/1 0 0 0 0 0 0 0 Fa0/2 0 0 0 0 0 0 0 Fa0/3 79569220 39013388 192542062 36797 0 0 0 Fa0/4 0 0 0 0 0 0 0 Fa0/5 0 0 0 0 0 0 0 Fa0/6 0 0 0 0 0 0 0 Fa0/7 0 0 0 0 0 3 0 Fa0/8 0 0 0 0 0 0 1 Fa0/9 0 0 0 0 0 0 0 Fa0/10 0 0 0 0 0 0 8062 Fa0/11 0 0 0 0 0 0 0 Fa0/12 0 0 0 0 0 0 1 Fa0/13 0 0 0 0 0 0 0 Fa0/14 0 0 0 0 0 0 0 Fa0/15 0 0 0 0 0 0 0 Fa0/16 0 0 0 0 0 0 0 Fa0/17 0 0 0 0 0 0 0 Fa0/18 0 0 0 0 0 0 0 Fa0/19 0 0 0 0 0 0 0 Fa0/20 0 0 0 0 0 0 0 Fa0/21 0 0 0 0 0 0 0 Fa0/22 0 0 0 0 0 0 0 Fa0/23 0 0 0 0 0 0 0 Fa0/24 0 0 0 0 0 0 0 Gi0/1 448854 267509 563956 302 0 0 0 Gi0/2 1262 664 3598 0 0 0 0
show process cpu
CPU utilization for five seconds: 5%/0%; one minute: 5%; five minutes: 5% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 0 8 0 0.00% 0.00% 0.00% 0 Chunk Manager 2 331833 31121683 10 0.00% 0.00% 0.00% 0 Load Meter 3 509 1339884 0 0.00% 0.00% 0.00% 0 DHCPD Timer 4 168749632 24332472 6935 0.00% 0.07% 0.05% 0 Check heaps 5 7701613 2593475 2969 0.15% 0.01% 0.00% 0 Pool Manager 6 0 1 0 0.00% 0.00% 0.00% 0 DiscardQ Backgro 7 0 2 0 0.00% 0.00% 0.00% 0 Timers 8 9 270 33 0.00% 0.00% 0.00% 0 WATCH_AFS 9 0 1 0 0.00% 0.00% 0.00% 0 HRPC asic-stats 10 0 1 0 0.00% 0.00% 0.00% 0 Crash writer 11 0 1 0 0.00% 0.00% 0.00% 0 Exception contro 12 0 1 0 0.00% 0.00% 0.00% 0 ARP Input 13 78657 161504522 0 0.00% 0.00% 0.00% 0 ARP Background 14 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT 15 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager 16 8 3 2666 0.00% 0.00% 0.00% 0 Entity MIB API 17 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage 18 30289 31072050 0 0.00% 0.00% 0.00% 0 IPC Event Notifi 19 463 2593313 0 0.00% 0.00% 0.00% 0 IPC Dynamic Cach 20 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv 21 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager 22 40337 151213854 0 0.00% 0.00% 0.00% 0 IPC Periodic Tim 23 35993 151213854 0 0.00% 0.00% 0.00% 0 IPC Deferred Por 24 0 1 0 0.00% 0.00% 0.00% 0 IPC Process leve 25 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager 26 1531 8888327 0 0.00% 0.00% 0.00% 0 IPC Check Queue 27 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat RX Cont 28 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat TX Cont 29 4290 15560845 0 0.00% 0.00% 0.00% 0 IPC Keep Alive M 30 6562 31072426 0 0.00% 0.00% 0.00% 0 IPC Loadometer 31 0 3 0 0.00% 0.00% 0.00% 0 PrstVbl 32 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
show process mem
Processor Pool Total: 21831860 Used: 16535464 Free: 5296396 I/O Pool Total: 4194304 Used: 2370532 Free: 1823772 Driver te Pool Total: 1048576 Used: 40 Free: 1048536 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 19719052 3248840 13682576 0 0 *Init* 0 0 12044 272416 12044 0 0 *Sched* 0 0 28935876 27665384 1277852 2617230 263240 *Dead* 0 0 0 0 394528 0 0 *MallocLite* 1 0 1544 1544 7160 0 0 Chunk Manager 2 0 232 232 4160 0 0 Load Meter 3 0 0 0 7032 0 0 DHCPD Timer 4 0 4652 232 11632 0 0 Check heaps 5 0 726593020 726746236 42192 1851459116 1851504332 Pool Manager 6 0 0 0 7160 0 0 DiscardQ Backgro 7 0 232 232 7160 0 0 Timers 8 0 0 0 4160 0 0 WATCH_AFS 9 0 0 0 7160 0 0 HRPC asic-stats 10 0 0 0 25160 0 0 Crash writer 11 0 340 0 7500 0 0 Exception contro 12 0 0 0 7160 0 0 ARP Input 13 0 688 688 7160 0 0 ARP Background 14 0 0 0 7160 0 0 AAA_SERVER_DEADT 15 0 0 0 13160 0 0 Policy Manager 16 0 24332 7564 23928 0 0 Entity MIB API 17 0 0 0 7160 0 0 IFS Agent Manage 18 0 0 0 7160 0 0 IPC Event Notifi 19 0 0 0 7160 0 0 IPC Dynamic Cach 20 0 0 0 7160 0 0 IPC Session Serv 21 0 0 0 7160 0 0 IPC Zone Manager 22 0 0 0 7160 0 0 IPC Periodic Tim 23 0 0 0 7160 0 0 IPC Deferred Por 24 0 0 0 7160 0 0 IPC Process leve 25 0 1688 0 8848 0 0 IPC Seat Manager 26 0 0 0 7160 0 0 IPC Check Queue 27 0 0 0 7160 0 0 IPC Seat RX Cont 28 0 0 0 7160 0 0 IPC Seat TX Cont 29 0 0 0 7160 0 0 IPC Keep Alive M 30 0 0 0 7160 0 0 IPC Loadometer
- Labels:
-
LAN Switching
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2018 02:45 PM
Hello,
first of all, make sure you got the basics covered. You have collisions on some interfaces, those alone can slow down the switch considerably. Nothing should be half duplex, everything should be full duplex.
GigabitEthernet0/1 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is 001f.279a.9f99 (bia 001f.279a.9f99) Description: InetNVT5 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 1/255, rxload 2/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex, 100Mb/s, media type is 10/100/1000BaseTX
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2018 03:11 PM
Hi,
For starter, your Gi0/1 interface is half-duplex/100mb and there are a lot of collisions and late collisions. You need to ensure Gi0/1 is running full-duplex/100.
There are a lot of error on your Fast Ethernet ports. You need to check the duplex/speed and cabling/nic.
Thanks
John
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2018 07:32 PM
I noticed that half-duplex setting. The appliance that this switch is plugged into has multiple NIC's on it, and they're tied to separate VLANs. Basically, the appliance is an internet service aggregator and so it connects our LAN to 2+ ISP's. The two half-duplex connections are that appliance, but only the one in the Gig ports (which were previously in the FastEth ports and exhibited the exact same behavior) are actually affected. The secondary ISP runs at full speed.
FWIW, the appliance says "100-Full Duplex", but according to the switch, that's not what it's actually doing. I didn't realize that until after I took these grabs. When I changed the Gig connection to "Auto Detect" on the appliance the problems went away, but simultaneously the console connection slowed to a crawl.
I don't have access to the switch this weekend to pull more data, but I'll try to grab that Monday before I swap out the switch.
Besides lots of framing errors, and a slow console connection, the only measurable behavior that I can actually document is speed tests on the 40/40mbps come back at 6mbps down, 35mbps up. The other ISP that is also half-duplex, is a 25/5mbps, and the speed tests read almost exactly that. Aside from that, I'm having weird, glitchy timeout issues on some SQL and proprietary TCP connections that are running between my VMWare hosts and the DMZ leg of the firewall.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2018 11:50 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-28-2018 03:58 PM - edited 12-28-2018 03:59 PM
Gi 0/1 & Fa 0/3 are both running on Half-Duplex. Look at the amount of collision for Fa 0/3 and compare that value with Gi 0/1.
If I remembered the behaviour of a 2950, half-duplex errors should be spamming the logs constantly.
Post the complete output to the command "sh interface Fa 0/3 controll".
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-31-2018 12:23 PM
Output from sh int fa 0/3 controller
FastEthernet0/3 is down, line protocol is down (notconnect)
Hardware is Fast Ethernet, address is 001f.279a.9f83 (bia 001f.279a.9f83)
Description: InetNVT3
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 1y8w, output 1w2d, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 8608
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
9488317029 packets input, 6710244949295 bytes, 0 no buffer
Received 5221 broadcasts (1049 multicasts)
0 runts, 0 giants, 0 throttles
1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1049 multicast, 0 pause input
0 input packets with dribble condition detected
9243040178 packets output, 6168937213911 bytes, 0 underruns
0 output errors, 118619405 collisions, 1 interface resets
7 unknown protocol drops
0 babbles, 192542062 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit FastEthernet0/3 Receive
1364176855 Bytes 1506032943 Bytes
448785383 Unicast frames 898377216 Unicast frames
17771682 Multicast frames 1049 Multicast frames
6122 Broadcast frames 4172 Broadcast frames
0 Too old frames 1505649430 Unicast bytes
151955080 Deferred frames 91402 Multicast bytes
0 MTU exceeded frames 291744 Broadcast bytes
79569220 1 collision frames 0 Alignment errors
17457768 2 collision frames 1 FCS errors
5775046 3 collision frames 0 Oversize frames
3400428 4 collision frames 0 Undersize frames
2645405 5 collision frames 0 Collision fragments
2222959 6 collision frames
2210691 7 collision frames 1410039121 Minimum size frames
2058562 8 collision frames 1439453258 65 to 127 byte frames
1672889 9 collision frames 1663826970 128 to 255 byte frames
1073288 10 collision frames 601915795 256 to 511 byte frames
319078 11 collision frames 281894942 512 to 1023 byte frames
105369 12 collision frames 4091186944 1024 to 1518 byte frames
41903 13 collision frames 0 Overrun frames
19492 14 collision frames 0 Pause frames
10510 15 collision frames
36797 Excessive collisions 0 Symbol error frames
192542062 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 0 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
1963333898 64 byte frames 0 Valid frames, too small
1182820777 127 byte frames
1400428239 255 byte frames 0 Too old frames
726202400 511 byte frames 0 Valid oversize frames
448661982 1023 byte frames 0 System FCS error frames
3521592882 1518 byte frames 0 RxPortFifoFull drop frame
0 Too large frames
79569220 Good (1 coll) frames
39013388 Good (>1 coll) frames
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-31-2018 02:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-31-2018 03:36 PM
@ZackC wrote:
Output from sh int fa 0/3 controller
FastEthernet0/3 is down, line protocol is down (notconnect)
Last input 1y8w, output 1w2d, output hang never
This is a red herring everyone.
Port has been down for > 1 year.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide