Switch is passing traffic but has lost management IP connectivity

[dbuckley77]

We have a 2950G that has lost IP connectivity via the management IP.  All ports, access and network trunks, are up and passing traffic but there we cannot reach via the management IP. Had been working previously and has gone up/down a few times.  Please advise.

[Peter Paluch]

Hello,

I am afraid that you will need to use a console port to troubleshoot this problem.

The obvious questions would be, after consoling into the switch:

1. Is the SVI with the management IP address in the up/up state?
2. Is the default gateway (show ip default-gateway - hidden command in older IOSes) configured correctly?
3. Can you ping the default gateway from the switch?
4. Are there perhaps all VTY connections exhausted? Check using show users how many users are currently logged in.
5. Any possibly relevant logging messages in show logging?

Best regards,
Peter

[dbuckley77]

The SVi is up/up

gateway is correct, the switch was accessible and now it's not but nothing has changed in the config

cannot ping gateway, looks like layer 2 works but layer 3 is not working at all

vty lines not exhausted

checked the logs and nothing relevant

[Peter Paluch]

Hi,

This is interesting. Let's dig a little deeper:

1. Can the switch ping its own IP address?
2. Does the switch have the MAC address of the default gateway in its ARP table? Check using show ip arp
3. If the ARP entry is there, can you try clearing it using clear ip arp? Make sure it is gone, then try pinging the default gateway again, and whether it works or not, check again if the ARP entry has been learned.
4. Using the MAC address of the default gateway from the ARP table, can you check the output of show mac address-table and check whether the MAC address points out the correct interface?

Thanks!

Best regards,
Peter

[dbuckley77]

1.  The switch can ping it's own IP.

2.  The gateway mac address was in the arp table.

3.  If I clear the arp cache the pinging is intermittent as it was before.  Sometimes it pings the gateway and sometimes it fails.

4.  I see the mac address of the gateway being learned on many different vlansonthe switch but they all do have the correct physical interface which is a trunk between the switch and the other switch where the gateway is configured.

5.  I rebooted but did not solve the issue.

[burleyman]

I just had the exact same thing happen to a Catalyst 2690X switch for no apparent reason. I consoled in and found no issues. I rebooted the switch and it started working and has not had a problem since. Nothing in the logs showed issue. Very odd.

Mike

[Peter Paluch]

Hi Mike,

That's the worst case scenario indeed! Let's hope, though, we can revive the IP connectivity on this 2950 without reloading it.

One extra command I would like to see the output from: Would it be possible to pull out the output of show ip redirect from this box?

Best regards,
Peter

[Joseph W. Doherty]

Over the years, I have seen similar, i.e. device will continue to forward traffic, but loses some management abilities. NB: Although I've seen it, it tends to be very rare. (Most recently, several months ago, it happened on a 4500 chassis.)

Generally, as Mike notes, a reboot cures the issue.

However, as OP notes a reboot didn't solve the problem, you might be looking an underlying software or hardware problem. (The latter might be age related.)

[rishisemwal]

We have more than 40 cisco switches in the network that has lost IP connectivity via the management IP.  All ports, access, and network trunks, are up and passing data and voice traffic but we cannot ping some time all the access switches via the management IP.   It has been working fine from many years but last few weeks it has gone up/down a few times.  All the switches are reachable from core switch and it is reachable from the local network once we ping the any management IP of the switch. I do not understand why it is happening.