Cisco 9500 Stack Error Messages - Need help for troubleshoot.

manish suthar · ‎10-09-2017

Hello Everyone,

I have recently built 2 Cisco 9500 switches in a stack. Got an multiple errors on console screen.

Following are the errors messages:

Oct 10 08:10:03.829: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:10:04.959: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:10:04.959: %NIF_MGR-6-PORT_CONN_PENDING: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:10:06.830: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:10:06.830: %NIF_MGR-6-PORT_CONN_PENDING: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:10:07.960: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:10:07.960: %NIF_MGR-6-PORT_CONN_READY: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:10:09.830: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:10:09.830: %NIF_MGR-6-PORT_CONN_READY: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:10:13.961: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:10:13.961: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:10:18.833: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:10:18.833: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:10:19.963: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:10:19.963: %NIF_MGR-6-PORT_CONN_PENDING: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:10:21.833: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:10:21.833: %NIF_MGR-6-PORT_CONN_PENDING: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:10:22.964: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr:
Oct 10 08:10:22.964: %NIF_MGR-6-PORT_CONN_READY: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:10:24.791: %ARP-4-NULL_SRC_MAC: NULL MAC address from 10.80.53.132 on Vl702
Oct 10 08:10:24.834: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:10:24.834: %NIF_MGR-6-PORT_CONN_READY: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:10:28.965: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:10:28.965: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:10:33.836: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:10:33.837: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_
Oct 10 08:46:43.326: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:46:43.326: %NIF_MGR-6-PORT_CONN_READY: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:46:47.422: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:46:47.422: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:46:55.328: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:46:55.328: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:46:56.422: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:46:56.422: %NIF_MGR-6-PORT_CONN_PENDING: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:46:58.328: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:46:58.328: %NIF_MGR-6-PORT_CONN_PENDING: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:46:59.423: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:46:59.423: %NIF_MGR-6-PORT_CONN_READY: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:47:01.328: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:47:01.328: %NIF_MGR-6-PORT_CONN_READY: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:47:05.424: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:47:05.424: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:47:13.328: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:47:13.328: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:47:14.424: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:47:14.424: %NIF_MGR-6-PORT_CONN_PENDING: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:47:16.329: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:47:16.329: %NIF_MGR-6-PORT_CONN_PENDING: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:47:17.424: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:47:17.424: %NIF_MGR-6-PORT_CONN_READY: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:47:19.329: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:47:19.329: %NIF_MGR-6-PORT_CONN_READY: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:47:23.424: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:47:23.424: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:47:31.329: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:47:31.329: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:47:32.424: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:47:32.424: %NIF_MGR-6-PORT_CONN_PENDING: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:47:34.329: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:47:34.329: %NIF_MGR-6-PORT_CONN_PENDING: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.
Oct 10 08:47:35.424: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:47:35.425: %NIF_MGR-6-PORT_CONN_READY: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:47:37.329: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Pending to Ready.
Oct 10 08:47:37.329: %NIF_MGR-6-PORT_CONN_READY: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is READY.
Oct 10 08:47:41.426: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:47:41.426: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:47:49.329: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 2 R0/0: nif_mgr: PORT lpn:38 changed state from Ready to Timeout.
Oct 10 08:47:49.329: %NIF_MGR-6-PORT_CONN_TIMEOUT: Switch 2 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection has TIMEOUT: CONN_ERR_CONN_TIMEOUT_ERR
Oct 10 08:47:50.426: %NIF_MGR-6-PORT_CONNECTION_STATUS_CHANGE: Switch 1 R0/0: nif_mgr: PORT lpn:38 changed state from Timeout to Pending.
Oct 10 08:47:50.426: %NIF_MGR-6-PORT_CONN_PENDING: Switch 1 R0/0: nif_mgr: Port 0, LPN 38 on front side stack link 0 connection is PENDING.

Need inputs for troublehsooting:

Leo Laohoo · ‎10-09-2017

I recommend you raise a TAC Case for this. This platform (hardware and software) are still new. I am suspecting this is a bug.

manish suthar · ‎10-10-2017

ohhhk...

Evancise · ‎05-01-2018

Is this bug fixed

I use firmware 16.8.1 , this bug still exist

> <!!

andrewsmith159159 · ‎05-14-2018

Hi,

I had a similar issue. We run 2 x 9500 40X switches on 16.8.1a and the stackwise refused to work.

I raised a TAC with Cisco who advised to change the rommon MAC_ADDR variable to include a 00 as it was missing from the front (eg :AB:CD:EF:GH:IJ). Once I did this the stacking working immediately after a reboot.

I'm still confirming why I had to change the rommon MAC_ADDR variable however early indications tell it could be related or explained by CSCvj43609

Hope this helps.

Andrew

Evancise · ‎05-14-2018

Hi Andrew

Can you show how to change rommon mac address

thanks

Will Cisco promise release new firmware to fix bug ??

andrewsmith159159 · ‎05-14-2018

Hi,

Reload your 9500 switch and boot into rommon. To do his push the stat button constantly (or whatever it's called nowadays) as it's booting up until you get the "switch:" prompt. It can take up to a minute.

Type "set" to list the rommon variables. You will see the MAC_ADDR variable which you need to change by typing:

"set MAC_ADDR=00:<rest of MAC ADDRESS that's already in there>"

then type "set" to see your changes and confirm the MAC address has the leading 00

Then Reload your switch.

Cheers,

Andrew

Evancise · ‎05-22-2018

Hi Andrew

Thanks for your guidance

The problem is solved

tanner.zaitt · ‎07-04-2021

Hello all, I can confirm CSCvj43609.
We find it in our "C9500-40X" Fuji 16.9.3.
Cisco TAC team also confirmed it.
They recommended upgrading the switch to the latest version where the issue has been fixed.
Best Regards.

tanner.zaitt · ‎07-04-2021

Do you have issues with traceroute mac ip ?