cisco MDS 9124 - Kernel rebooting?

[Jamie Hancock]

Hi,

We have a few  cisco MDS 9124 switches some of which seem to reboot there kernel at various time during the day.

Has anyone experienced this before?

I have pasted a few outputs below:-

llswL1san3# sh log nvram
2008 Aug 29 02:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 03:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 04:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 05:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 06:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 07:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 08:10:12 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 1 failed or shutdown (Serial number PAC11010L5D)
2008 Aug 29 08:10:13 llswL1san3 %PLATFORM-2-PS_FANFAIL: Fan in Power supply 1 failed
2008 Aug 29 08:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 09:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 10:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 11:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 12:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 13:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 14:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 15:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 16:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 17:50:57 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 18:51:02 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 19:51:02 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 20:51:02 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 21:51:02 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 29 22:51:02 llswL1san3 %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
2008 Aug 30 07:17:47 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 1 failed or shutdown (Serial number PAC11010L5D)
2008 Aug 30 07:17:47 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2008 Aug 30 07:17:47 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2008 Aug 30 07:17:47 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2008 Aug 30 07:17:47 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2008 Aug 30 07:17:47 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2008 Aug 31 23:01:32 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2008 Aug 31 23:01:32 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2008 Dec 19 08:36:50 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 2 failed or shutdown (Serial number PAC102803CG)
2008 Dec 19 08:36:51 llswL1san3 %PLATFORM-2-PS_FANFAIL: Fan in Power supply 2 failed
2008 Dec 22 00:38:47 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 1 failed or shutdown (Serial number PAC11010L5D)
2008 Dec 22 00:38:47 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2008 Dec 22 00:38:47 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2008 Dec 22 00:38:47 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2008 Dec 22 00:38:47 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2008 Dec 22 00:38:47 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2008 Dec 22 00:40:54 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2008 Dec 22 00:40:54 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2009 Apr  2 01:02:42 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2009 Apr  2 01:02:42 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2009 Apr  2 01:02:42 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2009 Apr  2 01:02:42 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2009 Apr  2 01:02:42 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 Apr  2 01:02:42 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 Apr  2 01:02:42 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 May 10 04:39:35 llswL1san3 %LC_PORT_CHANNEL-SLOT1-0-HW_FAILURE: fatal error in configureing B_MOD table: Invalid argument
2009 Aug 21 09:39:48 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 2 failed or shutdown (Serial number PAC102803CG)
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-PS_UNSUPPORTED: Detected an unsupported power supply 1 Not Installed for CISCO MDS 9500/9200 Multilayer Switches (Serial number PAC11010L5D)
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 Aug 23 01:58:37 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 Aug 24 01:38:41 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 2 failed or shutdown (Serial number PAC102803CG)
2009 Aug 24 01:40:36 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2009 Aug 24 01:40:36 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2009 Aug 24 01:40:36 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2009 Aug 24 01:40:36 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2009 Aug 24 01:40:36 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 Aug 24 01:40:36 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2009 Aug 24 01:40:36 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Mar 19 19:43:41 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 2 failed or shutdown (Serial number PAC102803CG)
2010 Mar 19 19:43:42 llswL1san3 %PLATFORM-2-PS_FANFAIL: Fan in Power supply 2 failed
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-PS_UNSUPPORTED: Detected an unsupported power supply 1 Not Installed for CISCO MDS 9500/9200 Multilayer Switches (Serial number PAC11010L5D)
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Mar 19 20:46:44 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Apr  3 01:12:03 llswL1san3 %PLATFORM-2-PS_FAIL: Power supply 2 failed or shutdown (Serial number PAC102803CG)
2010 Apr  3 01:12:04 llswL1san3 %PLATFORM-2-PS_FANFAIL: Fan in Power supply 2 failed
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-PS_UNSUPPORTED: Detected an unsupported power supply 1 Not Installed for CISCO MDS 9500/9200 Multilayer Switches (Serial number PAC11010L5D)
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Apr  3 04:59:03 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Aug  9 11:47:44 llswL1san3 %KERN-2-SYSTEM_MSG: Starting kernel... - kernel
2010 Aug  9 11:47:44 llswL1san3 %KERN-1-SYSTEM_MSG:  Entered kgdb_console_init:1960 - kernel
2010 Aug  9 11:48:15 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2010 Aug  9 11:48:15 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2010 Aug  9 11:48:15 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2010 Aug  9 11:48:15 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2010 Aug  9 11:48:15 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok
2010 Dec 15 02:57:15 llswL1san3 %KERN-2-SYSTEM_MSG: Starting kernel... - kernel
2010 Dec 15 02:57:15 llswL1san3 %KERN-1-SYSTEM_MSG:  Entered kgdb_console_init:1960 - kernel
2010 Dec 15 02:57:39 llswL1san3 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number PAC11010L5D)
2010 Dec 15 02:57:39 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok
2010 Dec 15 02:57:39 llswL1san3 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number PAC102803CG)
2010 Dec 15 02:57:39 llswL1san3 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2010 Dec 15 02:57:39 llswL1san3 %PLATFORM-2-FAN_OK: Fan module ok

llswL1san3# sh system reset-reason
----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---
1) No time
    Reason: Unknown
    Service:
    Version: 4.1(3a)
2) At 200533 usecs after Mon Aug  9 11:46:25 2010
    Reason: Reset by installer
    Service:
    Version: 3.1(3a)
3) No time
    Reason: Unknown
    Service:
    Version: 3.1(3a)
4) No time
    Reason: Unknown
    Service:
    Version: 3.1(3a)

Hope you can pool some light on this for me.

J.

[Michael Brown]

There are several causes that result in this error.  You should open a case with your MDS service provider and let them diagnose if this is a hardware or a software issue.  Unfortunalty, theses events can occur due to either one and the syslog output looks exactly the same in both cases.  The support engineers may be able to help identify if an upgrade to the software would help or not.

There was an issue like this where only 1 PS being installed could result in a reboot, but from what I can tell, you have 2 PS installed, and the s/n are not in the affected range.

-Mike