Cisco 9500 - Network Essentials not passing/responding to L3 Traffic

[alancelliott]

Hey all,

Got a Cisco 9500 as a new core switch. Went to cable it over this evening and ran into an issue where it wouldn't pass L3 traffic, or seemingly send or respond to ARPs (I do see some entries, but nothing from hosts I'm generating traffic from/to). The interfaces are up/up looking at the transceivers it looks like the optics are within spec on the sfp+'s. 

The spanning-tree topology builds. I can see Mac addresses on one of the trunk links, but the trunk links to our firewalls have empty Mac tables despite the interfaces being in an up state. IP routing is enabled, also the RIB is populated with the Subnets on the VLANs and the default route to the firewall VIP.

Pinging the VLAN interface of the directly connected switch fails, in both directions. I see some ARP entries, but pinging those hosts fails as well. I'm pretty stumped because it looks like a lower layer problem, but the up/up status and the interface counters not showing any problems is contrary to that thought. Has anyone seen behaviour like this before? Is it a bad switch? Can bad SFP's present like this? Any help would be great.

Cheers,

Alan

P.S. I did clear ARP and mac address table entries on other switches/hosts to no avail.

[balaji.bandi]

At this hard to say what is wrong here, we need more information like below :

1. what is the model of the switch

2. what code it running. ( IOS XE)

3.show some config of the interface and test outcome. (which can help to understand the issue)

4. give a description where the PC connected and where is uplink going to?

5. is this single switch or part of Stackwisevirtual

[alancelliott]

Hi Balaji, thanks for the reply,

1. C9500-48X-A.

2. 16.9.0

3. And 4.

I had to revert my change back to the old core last night since the 9500 wasn't working as expected, I will try and get some config later today. But my switch ports are all trunk links, my upstream is to a pfsense firewall and downstream is to a pair of sg500's. The link was up/up between the pfsense and 9500 but no mac addresses were being learned. The link to the sg500's was up/up non-LAG'd so one port was blocked by stp, the other one was forwarding and had Mac addresses. At the very least I expected to be able to ping from the mgmt svi on the sg500 to the svi on the same VLAN on the 9500, but that wasn't happening despite the up/up status and RIB being populated.

5. Non-stackwise the essentials lisence doesn't support stacking. It does support vrrp, and we will be getting another core switch for redundancy so I had vrrp configured, the show vrrp output said the switch was the master, but I even removed the vrrp config from the svi just to see if that was the problem, but still wouldn't work. Some of the SVIs have ACLs attached, so I even tried an permit IP any any to see if ACLs were an issue, but again to no avail.

Thanks again,

Cheers,

Alan

[balaji.bandi]

Thanks for the information, what was the change about, you have not mentioned in the original post, the upgrade change took place.

what was the version you trying to upgrade, the version you posted Older version after roll back or after upgraded version.

STP Blocking was there before upgrade - if not i will investigate that as part of suspect problem.

( we Clearly understand Essential do not support - SVL) - but rather assumption worth checking  - so thanks for the clarification.

The most important information we have now is, this was reported the upgrading taking place. have you read the complete release notes and any caveats you noticed part of this problem?

I would expect Cat 9500  to be root bridge for all the VLAN, can you post spanning-tree output along with configuration when you get a chance.

EDIT: Other notes i missed here, after the upgrade and booted, do you see any errors that ignored your old config?

[alancelliott]

Hey Balaji,

Thanks again for the quick reply. Sorry I haven't grabbed some config yet, the Mac I had my screen share setup on for the console session has stopped working, so I'll fix it on Monday.

I am making an Architecture change to the network. When I was hired as the Network admin all the switches are small business switches. I was asked to get 10Gb speed from our servers, so this is an upgrade to a proper 10GE core switch. I'm also moving L3 in from the edge to the core. The firewall previously had been handling it all. With that said I expected ARP and Mac table to cause some issues since the Mac addresses for the L3 interfaces would be changing. So I did clear dynamic Mac and all ARP entries on the other switches in the infrastructure, as well as the ARP entries from hosts I was testing with. Still the C9500 wouldn't respond to ICMPs or send ICMPs. I did have a debug ARP running on the 9500 while  sending pings but there was no output at all, which makes me think something may be wrong with the switch.

Cheers,

Alan

[balaji.bandi · Accepted solution]

we need more inputs as I have asked, have many Cat 9500 never seen the behaviour as you mentioned. until I missing some information here.

Also, try to upgrade to the latest stable version of IOS XE and deploy again, 16.9.X  is old - also some MST instance issue reported.

[alancelliott]

Hey,

Just an update I was able to update the firmware to 16.12.03a and it immediately fixed the issue I was seeing. Super weird, but maybe was related to the bug you previously specified since I was using MST in our environment. I did verify the STP status on the trunk ports, and the interface was forwarding for my VLANs. So some weird behaviour, but thanks Balaji for helping, and hope it helps someone else having similar issues.

Cheers,

Alan

[balaji.bandi]

Thants good news, glad all went well and it was usefull  the  information i have provided.