4500X Routing/ARP Issue

[John King]

We are experiencing an odd issue with one of our Catalyst 4500X switches.

We recently installed a new one of these into one of our campuses as a replacement for the Core switch of the site. We had already installed one into the Core of another site with no issues.

They are running different versions

Working site on:

Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.07.00.E RELEASE SOFTWARE (fc4)

Issue site:

Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.09.00.E RELEASE SOFTWARE (fc1)

The problem we are experiencing is a loss of connectivity *through* the switch. For example, it initially came up that our monitoring software could not ping through to certain access switches on the network. Now we are having an issue with the print server not being able to contact printers. It seems that it is all contact with the outlying devices, as for the printers, once it goes out of contact, you cannot access the webpage, ping, or print to the device at all.

Restarting the outlying device (the printer) will restore functionality for it. If you ping the device *from* the 4500X, it comes back up immediately. This is what led me to think it may be an ARP table issue.

It is a sporadic issue, but is troublesome when it does occur. Is there something I can do to determine the cause or troubleshoot further? Has anyone seen this before?

[jerrymatson1]

Since this situation is developing...

We have found that the problem may be changing with network utilization. That's not to say that the links are saturated by any means. Yesterday, the office in question was closed for a holiday and we could not reproduce the problem. Today, with the office open, we can easily reproduce it not only on indirectly connected switches but with even the directly connected as well.

I am downloading a few versions of code to the switch so that we could potentially switch back and forth. We're waiting on a TAC engagement at this point but I'll report back once we know more.

[jerrymatson1]

Quick update on this issue:

Our first bug was a flood set (show platform hardware floodset vlan <vlan id> and show platform software floodset vlan <vlan  id>) mismatch.

After moving to a newer version of code: cat4500e-universalk9.SPA.03.09.02.E.152-5.E2.bin

We then ran in to another bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvc08452

High cpu due to EPC WS Pkt Send pro on IOS-XE

This bug manifests with a process (iosd) driving one of the CPUs to 100% utilization. IOSD is an older IOS daemon running in IOS-XE and can only use a single processor. The issue here is that ARP packets were dropping due to process utilization.

TAC did some research and we're now loading the following code: cat4500e-universalk9.SPA.03.08.06.E.152-4.E6.bin

This code is said to remedy both problems and be free of any other major bugs.

Time will tell, but working to implement this code at this time.

[zatlawey1]

hi,

is the problem solved?

[vsinharockwell]

We are having the similar problem and we saw that the MAC address of VLAN on 4500 is being spoofed. We saw that all users in a specific VLAN started complaining that they cant connect. investigation identified that their gateway MAC is being learnt from an end port where we had a Zebra device connected. when we did the shut /no-shut to that interface, that MAC disappeared and all users on that VLAN could access the network. We are investigating that Zebra device to find details.