Cisco 2950 loss connectivity management Vlan

spinning2008 · ‎04-09-2012

Hi all,

Randomly when I try to access to 2950 from management tools, switch is unreachable, I have to access from other switch and reload 2950.

Problem only is from managemt tool to managament vlan 1 2950.

The strange thing is that management interface is encountering a very fast increase of throttles, broadcast and ignored packets:

2950#show interfaces vlan 1

Vlan1 is up, line protocol is up

Hardware is CPU Interface, address is 0023.3488.fd65 (bia 0023.3488.fd65)

Internet address is xxxxxxxxxx

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output never, output hang never

Last clearing of "show interface" counters 18:01:29

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 0 drops

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 75000 bits/sec, 74 packets/sec

30 second output rate 20000 bits/sec, 19 packets/sec

3649696 packets input, 444766538 bytes, 0 no buffer

Received 2146441 broadcasts (0 IP multicast)

0 runts, 0 giants, 17226 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 1496048 ignored

1276594 packets output, 211444551 bytes, 0 underruns

0 output errors, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

show buffers

Buffer elements:

499 in free list (500 max allowed)

163971601 hits, 0 misses, 0 created

Public buffer pools:

Small buffers, 104 bytes (total 33, permanent 25, peak 181 @ 6w5d):

33 in free list (20 min, 60 max allowed)

191360417 hits, 4012 misses, 7336 trims, 7344 created

26 failures (0 no memory)

Middle buffers, 600 bytes (total 30, permanent 15, peak 76 @ 6w5d):

28 in free list (10 min, 30 max allowed)

11560828 hits, 3320 misses, 3269 trims, 3284 created

1562 failures (0 no memory)

Big buffers, 1524 bytes (total 9, permanent 5, peak 25 @ 5w6d):

9 in free list (5 min, 10 max allowed)

15740378 hits, 4396 misses, 3719 trims, 3723 created

2344 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 10, permanent 0, peak 24 @ 6w4d):

10 in free list (0 min, 10 max allowed)

147538 hits, 1870 misses, 1808 trims, 1818 created

986 failures (0 no memory)

Large buffers, 5024 bytes (total 1, permanent 0, peak 6 @ 6w5d):

1 in free list (0 min, 5 max allowed)

123 hits, 863 misses, 600 trims, 601 created

863 failures (0 no memory)

Huge buffers, 18024 bytes (total 1, permanent 0, peak 4 @ 5w6d):

1 in free list (0 min, 2 max allowed)

60 hits, 803 misses, 595 trims, 596 created

803 failures (0 no memory)

Interface buffer pools:

Calhoun Packet Receive Pool buffers, 1560 bytes (total 512, permanent 512):

447 in free list (0 min, 512 max allowed)

310549933 hits, 0 misses

show buffers failures

Caller Pool Size When

0x802D7634 Large 170 08:07:37

0x802D7634 Huge 170 08:07:37

0x802D7634 Middle 170 07:07:37

0x802D7634 Middle 170 05:33:54

0x802D7634 Middle 170 03:56:43

0x802D7634 Middle 170 00:12:26

0x802D7634 Middle 178 00:12:26

0x802D7634 Middle 170 00:12:26

2950#show region address 0x802D7634

Address 0x802D7634 is located physically in :

Name : text

Class : IText

Media : R/W

Start : 0x80010000

End : 0x8056FA63

Size : 0x0055FA64

It could be a hardware problem?, IOS has been upgraded.

Best Regards.

glen.grant · ‎04-09-2012

If it's been a long time since it was reloaded you might want to think about that . If you need to know what the broadcasts are etc.. then you will have to span a port and wireshark it ... to see what is going on .

bbaillie · ‎04-09-2012

Hi,

How many hosts are in VLAN 1? The last time I saw symptoms like this was on a very flat network with over 2500 hosts on VLAN 1. The big tipoff is the the volume of broadcast traffic. The management interface must check each broadcast to see if it is the intended recipient, and then discard the packet if it is not the intended recipient. This means the management interface is occupied with packet receipt/discard and not performing the intended response to management traffic.

The network most likely has redundant links and a bit of spanningtree BPDU skew happening as well, check the spanning tree root for Topology changes and verify the spanning tree radius it's probably at the upper limit. Twenty bit mask or smaller in a switched network, never a good scene.

Cheers,

Brian

spinning2008 · ‎04-11-2012

Thanks for reply,

more or less are 1000 hosts in Vlan 1, but the big problem is if I ping from switch to Managament PC I can´t ping, but if I do clear arp I can, in few minutes after the clear arp I can't ping again.

It could be a duplicated mac?

Regards.

glen.grant · ‎04-11-2012

Could be a duplicate ip address . Check at the L3 end for macs and see if it matches the 2950 or someone else.

spinning2008 · ‎04-11-2012

From switch to switch I can ping but from PC to switch or switch to PC I can`t, yes it's ok, I did trace mac and matches the 2950.

bbaillie · ‎04-11-2012

Hi,

If the network is smaller than 1000 then you have either a subnet mask problem on the PC or the switch, alternately you could have a host doing a proxy arp response and the PC or switch is recieving this bogus proxy arp request.

I would not suspect a duplicate mac, I would lean more towards a duplicate IP address on the wire, which means the IP address of the switch is duplicate. Check the PC arp cache when ping is OK then again when the ping fails.

Cheers,

spinning2008 · ‎04-12-2012

Hi again,

I changed the managament Ip address of the Vlan 1 in the switch and the problem is still there. I don't know if there is hardware problem with the switch???

Thanks for your help.

glen.grant · ‎04-12-2012

Have you reloaded the switch seems I remember the 2950's having memory fragmentation issues in the earlier IOS versions and a reload would fix it at least for awhile. Do the uplink ports look ok with no interface errors etc. . What version is the 2950 running ?

bbaillie · ‎04-12-2012

Hi,

Did you check the arp cache on the PC when ping was OK and when ping failed to determine if the MAC address was different for the switch?

Also just for a proof of concept, try the command "show spanning-tree blockedports". Then if you spot some ports that are in blocking mode, do a "shutdown" on each one of them, they are redundant paths so no risk of an outage. Watch to see if the switch remains responsive or goes non responsive again.

After the test of course do a "no shutdown" on the ports and again see if the switch behaves as badly as before.

Cheers,

spinning2008 · ‎04-15-2012

ok I will try. A possible solution could be put arp timeout 300 sec in VLan1?

Regards

bbaillie · ‎04-15-2012

Hi,

I don't beleive adjusting the arp timeout on VLAN 1 will help much.

The first item is determine the root cause. This means we need to rule out the proxy-arp or duplicate IP suspicions.

This can be accomplished by examining the arp cache on the PC when ping is OK and when ping doesn't work. If this is ruled out, next take some workload off the switch to determine if this is load related, shutting down blocking ports will accomplish this.

Cheers,

rafafilho11 · ‎08-31-2018

I was with the problem on my network with 2950s, but after I tried a lot of things I figured out what was happens.

NO stp recalculation, no crc, process and memory issue.

I was not using vlan 1, but my management vlan was showing packet loss and high latency.

I put a machine on the same vlan as the management vlan and the machine not show the same problem as management of switch 2950.

The other switches on the same network seguiment was not showing packet loss or high latency.

Problem solved:

I allowed on the trunk just the vlans used on that switch.

switchport trunk allowed vlan x,y