04-19-2013 04:29 PM - edited 03-07-2019 12:55 PM
hi, i have a standalone SUP720, no traffic is passing through it. only one port connected for remote telnet, its kind of backup device.
today i checked its log, and there are some wiered message.
is it a software bug or my SUP has gone bad?
any suggestion/help would be appreciated.
*Apr 19 18:19:02: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:5
*Apr 19 18:19:03: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=7% RP=2% Traffic=0%
netint_thr_active[0], Tx_Rate[658], Rx_Rate[80], dev=4[IPv6, fail=5]
*Apr 19 18:20:29: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:10
*Apr 19 18:20:30: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[676], Rx_Rate[100], dev=4[IPv6, fail=10]
*Apr 19 18:21:56: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:15
*Apr 19 18:21:57: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=1% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[636], Rx_Rate[80], dev=4[IPv6, fail=15]
*Apr 19 18:23:23: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:20
*Apr 19 18:23:24: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=7% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[636], Rx_Rate[80], dev=4[IPv6, fail=20]
*Apr 19 18:24:50: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:25
*Apr 19 18:24:51: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=5% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[538], Rx_Rate[20], dev=4[IPv6, fail=25]
*Apr 19 18:26:16: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:30
*Apr 19 18:26:17: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=1% Traffic=0%
netint_thr_active[0], Tx_Rate[578], Rx_Rate[40], dev=4[IPv6, fail=30]
*Apr 19 18:27:43: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:35
*Apr 19 18:27:44: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=6% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[608], Rx_Rate[80], dev=4[IPv6, fail=35]
*Apr 19 18:29:10: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:40
*Apr 19 18:29:11: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=5% RP=2% Traffic=0%
netint_thr_active[0], Tx_Rate[670], Rx_Rate[100], dev=4[IPv6, fail=40]
*Apr 19 18:30:36: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:45
*Apr 19 18:30:37: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[608], Rx_Rate[80], dev=4[IPv6, fail=45]
*Apr 19 18:32:03: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:50
*Apr 19 18:32:04: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=2% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[698], Rx_Rate[100], dev=4[IPv6, fail=50]
*Apr 19 18:33:30: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:55
*Apr 19 18:33:31: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[658], Rx_Rate[80], dev=4[IPv6, fail=55]
*Apr 19 18:34:56: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:60
*Apr 19 18:34:57: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[653], Rx_Rate[100], dev=4[IPv6, fail=60]
*Apr 19 18:36:23: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:65
*Apr 19 18:36:24: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=4% RP=2% Traffic=0%
netint_thr_active[0], Tx_Rate[578], Rx_Rate[60], dev=4[IPv6, fail=65]
*Apr 19 18:36:58: %CONST_DIAG-SP-6-HM_TEST_RECOVERED: Module 6 TestSPRPInbandPing recovered after 66 consecutive failure(s)
*Apr 19 18:38:24: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:5
*Apr 19 18:38:25: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[653], Rx_Rate[100], dev=4[IPv6, fail=5]
*Apr 19 18:39:35: %CONST_DIAG-SP-6-HM_TEST_RECOVERED: Module 6 TestSPRPInbandPing recovered after 8 consecutive failure(s)
*Apr 19 18:42:08: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:5
*Apr 19 18:42:09: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=4% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[670], Rx_Rate[100], dev=4[IPv6, fail=5]
*Apr 19 18:43:34: %CONST_DIAG-SP-6-HM_TEST_RECOVERED: Module 6 TestSPRPInbandPing recovered after 9 consecutive failure(s)
04-19-2013 05:11 PM
Hi,
These messages are from Module 6. Is your sup in modules 6 or is this a line card?
What is the output of "sh module"?
HTH
04-19-2013 07:00 PM
yes its SUP in module 6
6506BK>sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
2 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45
3 48 SFM-capable 48-port 10/100 Mbps RJ45 WS-X6548-RJ-45
4 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
5 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45
6 2 Supervisor Engine 720 (Active) WS-SUP720-BASE
04-19-2013 07:50 PM
Make sure the supervisor card is properly inserted/seated on the slot.
If it is, then contact Cisco TAC and get the card RMA-ed.
04-22-2013 07:14 PM
Will do this excercise and update.
04-19-2013 08:37 PM
In addition to what Leo suggested, it maybe also helpful if you load a different version of IOS. It could be software issue.
HTH
04-22-2013 07:15 PM
I have same IOS running on my other 720 SUP/6506 with same cards and hardware. so i believe it might not be software issue?
04-19-2013 10:08 PM
Hi Ahmed,
This error message means that there was no communication between the
supervisor and the MSFC. This is most likely a hardware issue that could
however have been transient. The usual procedure is to monitor the router to
be sure it does not reoccur. If it happens again, then supervisor needs to
be RMA.
Whats the status of this module under show module?
Also plan for RMA of the module.
HTH
Regards
Inayath
*Plz rate all usefull posts.
04-22-2013 07:18 PM
Error keeps coming, they enver stops.
show module shows minor errors.
6506BK#sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
2 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45
3 48 SFM-capable 48-port 10/100 Mbps RJ45 WS-X6548-RJ-45
4 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
5 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45
6 2 Supervisor Engine 720 (Active) WS-SUP720-BASE
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0008.7dce.fae8 to 0008.7dce.fb17 10.1 7.2(1) 8.5(0.46)RFW Ok
2 0050.f0af.a174 to 0050.f0af.a1a3 1.1 4.2(0.24)VAI 8.5(0.46)RFW Ok
3 0008.7dcd.f9f0 to 0008.7dcd.fa1f 6.1 6.3(1) 8.5(0.46)RFW Ok
4 0013.1a0e.0d20 to 0013.1a0e.0d4f 10.1 7.2(1) 8.5(0.46)RFW Ok
5 00d0.bce8.aba0 to 00d0.bce8.abcf 1.1 4.2(0.24)VAI 8.5(0.46)RFW Ok
6 0011.21b5.6c34 to 0011.21b5.6c37 3.2 8.1(3) 12.2(18)SXF1 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
6 Policy Feature Card 3 WS-F6K-PFC3A 2.4 Ok
6 MSFC3 Daughterboard WS-SUP720 2.4 Ok
Mod Online Diag Status
---- -------------------
1 Minor Error
2 Pass
3 Minor Error
4 Minor Error
5 Pass
6 Minor Error
Canto do RMA, since ots not under RMA contract with Cisco :-s
04-22-2013 07:54 PM
Hi Ahmed,
If no contract for RMA. Then I would suggest to do the Hard Reset of the module on this switch and update me the result.
1- Remove all the modules which are showing Minor Error and insert it one by one.
2- Get the following outputs:
1- Show module
2- show logging
3- show diagnostic result module all.
HTH
Regards
Inayath
05-15-2013 01:35 PM
Hi Inayat,
Sorry i am late here , actually had to involve onshore field support to help me out.
i have taken out all modules, and now only SUP720 and one FastEthernet module present in the Switch.
i asked Field support to remove all Cards, and then put back Sup720 and one FastEthernet module.
here is output of your requested commands, i hope you can guid me in some direction to resolve this issue
sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
6 2 Supervisor Engine 720 (Active) WS-SUP720-BASE
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0008.7dce.fae8 to 0008.7dce.fb17 10.1 7.2(1) 8.5(0.46)RFW Ok
6 0011.21b5.6c34 to 0011.21b5.6c37 3.2 8.1(3) 12.2(18)SXF1 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
6 Policy Feature Card 3 WS-F6K-PFC3A 2.4 Ok
6 MSFC3 Daughterboard WS-SUP720 2.4 Ok
Mod Online Diag Status
---- -------------------
1 Minor Error
6 Minor Error
--------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------
6506BK#sh logging
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns)
Console logging: level debugging, 54 messages logged
Monitor logging: level debugging, 0 messages logged
Buffer logging: level debugging, 54 messages logged
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Trap logging: level informational, 45 message lines logged
Log Buffer (8192 bytes):
00:01:01: curr is 0x0
00:01:01: RP: Currently running ROMMON from S (Gold) region
*May 15 14:31:36: %SYS-6-CLOCKUPDATE: System clock has been updated from 19:31:36 UTC Wed May 15 2013 to 14:31:36 EST Wed May 15 2013, configured from console by console.
*May 15 15:31:36: %SYS-6-CLOCKUPDATE: System clock has been updated from 14:31:36 EST Wed May 15 2013 to 15:31:36 EDT Wed May 15 2013, configured from console by console.
*May 15 15:32:10: %SYS-5-CONFIG_I: Configured from memory by console
*May 15 15:32:11: %SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-M), Version 12.2(18)SXF17, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by cisco Systems, Inc.
Compiled Fri 25-Sep-09 10:53 by ccai
*May 15 15:32:11: %MFIB_CONST_RP-6-REPLICATION_MODE_CHANGE: Replication Mode Change Detected. Current system replication mode is Ingress
*May 15 15:32:12: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Firmware compiled 15-Jul-09 16:40 by integ Build [100]
*May 15 15:32:12: %OIR-SP-6-CONSOLE: Changing console ownership to route processor
*May 15 15:32:13: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*May 15 15:32:13: %FILESYS-SP-5-MISMATCH: bootflash device programming algorithm mismatch detected, reformat is recommended
00:01:09: SP: SP: Currently running ROMMON from S (Gold) region
*May 15 15:32:14: %SYS-SP-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) s72033_sp Software (s72033_sp-IPSERVICESK9_WAN-M), Version 12.2(18)SXF17, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by cisco Systems, Inc.
Compiled Fri 25-Sep-09 11:01 by ccai
*May 15 15:32:14: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*May 15 15:32:14: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
*May 15 15:32:14: %OIR-SP-6-INSPS: Power supply inserted in slot 2
*May 15 15:32:16: %C6KENV-SP-4-FANHIOUTPUT: Version 2 high-output fan-tray is in effect
*May 15 15:32:20: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 6 became active.
*May 15 15:32:20: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
*May 15 15:32:46: %DIAG-SP-3-MINOR: Module 6: Online Diagnostics detected a Minor Error. Please use 'show diagnostic result
*May 15 15:32:46: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 6: TestFibDevices failed
*May 15 15:32:46: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 6: TestIPv6FibShortcut failed
*May 15 15:32:46: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 6: TestL3VlanMet failed
*May 15 15:32:46: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 6: TestTransceiverIntegrity failed on port(s) 1
*May 15 15:32:46: %PM_SCP-SP-3-TRANSCEIVER_UNSUPPORTED: Unsupported transceiver in LAN port 6/1
*May 15 15:32:46: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 6: TestT
*May 15 15:32:46: %PM_SCP-SP-3-TRANSCEIVER_UNSUPPORTED: Unsupported transceiver in LAN port 6/2
*May 15 15:32:47: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
*May 15 15:32:49: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
*May 15 15:33:00: %DIAG-SP-3-MINOR: Module 1: Online Diagnostics detected a Minor Error. Please use 'show diagnostic result
*May 15 15:33:00: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 1: TestL3VlanMet failed
*May 15 15:33:01: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
*May 15 15:34:14: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:5
*May 15 15:34:15: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=9% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[698], Rx_Rate[100], dev=4[IPv6, fail=5]
*May 15 15:35:41: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:10
*May 15 15:35:42: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[658], Rx_Rate[80], dev=4[IPv6, fail=10]
*May 15 15:37:08: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:15
*May 15 15:37:09: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=3% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[698], Rx_Rate[100], dev=4[IPv6, fail=15]
*May 15 15:38:34: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:20
*May 15 15:38:35: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=4% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[556], Rx_Rate[40], dev=4[IPv6, fail=20]
*May 15 15:40:01: %CONST_DIAG-SP-3-HM_TEST_FAIL: Module 6 TestSPRPInbandPing consecutive failure count:25
*May 15 15:40:02: %CONST_DIAG-SP-6-HM_TEST_INFO: CPU util(5sec): SP=7% RP=0% Traffic=0%
netint_thr_active[0], Tx_Rate[538], Rx_Rate[20], dev=4[IPv6, fail=25]
*May 15 15:40:53: %CONST_DIAG-SP-6-HM_TEST_RECOVERED: Module 6 TestSPRPInbandPing recovered after 27 consecutive failure(s)
-------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------
6506BK# sh diagnostic result module all
Current bootup diagnostic level: minimal
Module 1: SFM-capable 48 port 10/100/1000mb RJ45
Overall Diagnostic Result for Module 1 : MINOR ERROR
Diagnostic level at card bootup: minimal
Test results: (. = Pass, F = Fail, U = Untested)
1) TestLoopback:
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
. . . . . . . . . . . . . . . . . . . . . . . .
Port 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
----------------------------------------------------------------------------
. . . . . . . . . . . . . . . . . . . . . . . .
2) TestSynchedFabChannel -----------> .
3) TestL3VlanMet -------------------> F
4) TestIngressSpan -----------------> .
5) TestEgressSpan ------------------> .
6) TestAsicMemory ------------------> U
7) TestFirmwareDiagStatus ----------> .
8) TestEobcStressPing --------------> U
Module 6: Supervisor Engine 720 (Active)
Overall Diagnostic Result for Module 6 : MINOR ERROR
Diagnostic level at card bootup: minimal
Test results: (. = Pass, F = Fail, U = Untested)
1) TestScratchRegister -------------> .
2) TestSPRPInbandPing --------------> F
3) TestTransceiverIntegrity:
Port 1 2
----------
F F
4) TestActiveToStandbyLoopback:
Port 1 2
----------
U U
5) TestLoopback:
Port 1 2
----------
. .
6) TestNewIndexLearn ---------------> .
7) TestDontConditionalLearn --------> .
8) TestBadBpduTrap -----------------> .
9) TestMatchCapture ----------------> .
10) TestProtocolMatchChannel --------> .
11) TestFibDevices ------------------> F
12) TestIPv4FibShortcut -------------> .
13) TestL3Capture2 ------------------> .
14) TestIPv6FibShortcut -------------> F
15) TestMPLSFibShortcut -------------> .
16) TestNATFibShortcut --------------> .
17) TestAclPermit -------------------> .
18) TestAclDeny ---------------------> .
19) TestQoSTcam ---------------------> .
20) TestL3VlanMet -------------------> F
21) TestIngressSpan -----------------> .
22) TestEgressSpan ------------------> .
23) TestNetflowInlineRewrite:
Port 1 2
----------
U U
24) TestFabricSnakeForward ----------> .
25) TestFabricSnakeBackward ---------> .
26) TestTrafficStress ---------------> U
27) TestFibTcamSSRAM ----------------> U
28) TestAsicMemory ------------------> U
29) TestNetflowTcam -----------------> U
30) ScheduleSwitchover --------------> U
31) TestFirmwareDiagStatus ----------> .
32) TestFabricFlowControlStatus -----> U
Thank you for Every one help and interest so far