Management IP of Switch Gets Stolen, No Matter What IP I Assign it to

[Dean Romanelli]

Hi All,

Bit of a strange issue. I have a site with 6 IDF's, housing Cisco 3560 POE switches, and each IDF is on a different subnet.  In two of those IDF's, the 3560 switches keep having their management IP's stolen. However, I do not believe it is an IP conflict causing the issue; I think it is the switches themselves.  Below is an explanation of one of the IDF's. The other one is having the same issue, but different rogue mac.

So from my core 6509 switch, which connects to all IDF's, under normal operation, my IDF switch at 192.168.48.230 is visible in MAC & ARP, and is pingable from 6509:

SWCoreMSDC-6509#show arp | inc 48.230
Internet 192.168.48.230 0 001f.ca74.1941 ARPA Vlan3

SWCoreMSDC-6509#show mac-add | inc 1941
* 3 001f.ca74.1941 dynamic Yes 5 Po7

SWCoreMSDC-6509#ping 192.168.48.230

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.48.230, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/8 ms

However, when I lose management to that switch at 192.168.48.230, I log back into the 6509 core switch and show arp on 192.168.48.230, and it now shows a different mac address:

SWCoreMSDC-6509#show arp | inc 192.168.48.230
Internet 192.168.48.230 43 6c88.1424.2e51 ARPA Vlan3

Then when I try to find out where it is coming from via the mac address table, there is no entry for the mac address, and the IP address no longer responds to pings.

SWCoreMSDC-6509#show arp | inc 192.168.48.230
Internet 192.168.48.230 43 6c88.1424.2e51 ARPA Vlan3

SWCoreMSDC-6509#show mac-add | inc 2e51
SWCoreMSDC-6509#
SWCoreMSDC-6509#
SWCoreMSDC-6509#

SWCoreMSDC-6509#ping 192.168.48.230

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.48.230, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Doing a clear ip arp in the 6509 core switch restores visibility, but I lose it again about 30 seconds later every time.

What is most puzzling, however, is that I got tired of fighting the issue and changed the management IP address of the switch to something else on the 48 subnet.  Maybe 20 minutes later that IP started doing the same thing.  So my suspicion is that the switch is acting up since I can't trace the mac-address, the IP doesn't ping when in a stolen state, and any management IP I assign to my switch is getting stolen, not just a specific one.

Has anyone come across anything like this?  I've not been able to find any resources online thus far to explain this.

Thanks.

[dukenuk96]

Hi

you may not see mac 2e51 because port with that MAC could have already gone down. I would recommend two things:

1. Your network equipment's management address should be in a separate VLAN to avoid interaction with user's or server's address space

2. Configure logging of ARP, MAC and port state changes - this will let you to see where that MAC appeared from.

[Dean Romanelli]

Hi,

Great idea on #1. I do have a vlan that is not distributed to end users. I will try that this weekend and report back the results.

Thanks.

[dukenuk96]

Go on, will wait for news

[Paul Chapman]

Hi Dean -

A lookup of the MAC address shows that 6c:88:14 is from an Intel NIC.  Are you sure that your management range is not overlapping your DHCP?

Also, on the remote switch, what does the log show?

PSC

[Dean Romanelli]

Hi Paul,

The management IP of the switch is on the same subnet as the DHCP pool but the .230 address is excluded from distribution.  I am going to change it to a different mgmt subnet and see if the problem goes away.