Cisco 9500-40X Layer 3 routing stop working

[rparres.sempere]

Hi you,

yesterday I had a big problem with my Cisco 9500, working in Stackwise-virtual, and the funcionality of Layer 3 routing. The case is that the devices were working well since 2019 but suddenly, this function stopped and the routing started to fail.

The situation is that the device can't ping to its own local interface but in the sh ip int b, the interface vlan is up in physical and protocol. In this momente the devices are working in layer2 but there is not way to do to work them in layer3. The only way to get access to the devices in serial connection

I have reviewed the stackwise-virtual (OK), the license (9500 Network Advantage) is perpetual, the firmware is 16.9.3 (is working the same all the time), the 'ip routing' command was configured, the routing table is showed with interface connected.

My doubt is about DNA. The devices were added to DNAC server and the license (9500-40X DNA Advantage) expired in 2022-SEPT-20 but it happent two days ago.

Anyone can have any idea why the layer 3 function is disable in this moment?

Thanks for your support and best regards

[Leo Laohoo]

Could be a memory leak which spilled over and affected other functionality. 

16.9.3 is not really that mature. 

If this pair has been online since 2019?  Have you checked the control-plane memory utilization lately? 

The command is "sh platform software status control-processor brief".  For something that has been up that long the memory utilization, ideally, should be <45%.

[rparres.sempere]

Hi Leo Laohoo,

thanks for replying.

The pair was working since 2019 and in all this time there is a few reboots, but the devices were working correctly with the l3. The night where we had issues, the pair was rebooted but the problem wasn't resolved so now the pair is working only as a l2.

We had 3 pair more in the ubication working well without layer 3 issues. For example, one pair has the next output that I have attached. All the 4 pair were installed in the same time and they have the same firmware but only one pair have had this problem.

What can I check more?

Thanks

[Leo Laohoo]

Make sure to constantly monitor that output.  Not every NMS will monitor it.  

For a beast like a 9400/9500/9600, the memory utilization should <25%.