How to run GOLD diagnostic tests on Catalyst 6500/6000 series switches to troubleshoot hardware issues

[TCC_2]

Core issue

This document discusses how to use the GOLD diagnostics online testing  feature to identify possible hardware issues and capture the test output for Technical Support  engineer review.

Online diagnostics are categorized as bootup, on-demand,  schedule, or health monitoring diagnostics. Bootup diagnostics run during bootup,  module Online Insertion and Removal (OIR), or switchover to a backup supervisor engine. On-demand diagnostics  run from the Command Line Interface (CLI). Schedule diagnostics run at user-designated intervals or  specified times when the switch is connected to a live network. Health-monitoring runs in the background.

This scenario displays examples using on-demand diagnostics to  troubleshoot possible hardware failures and captures the test results for Technical Support  review if a service request is required.

Resolution

When a switch component exhibits symptoms of possible hardware failure,  additional diagnostics tests should be performed to confirm the issue is hardware  related and rule out other causes of the symptoms.

Some tests disrupt network activity, so it is important to verify that the  test you run does not affect your current network traffic.

Runtime diagnostic checks can be run on demand or scheduled to run at a  specific time. They can run continually in the background. When a module or  supervisor exhibits failure symptoms or an error message is received, the test can  usually be performed on demand.

Issuing a diagnostic start command triggers on-demand diagnostics tests  statically. Specify how many times a test runs and whether to continue running  the test upon failure detection. On-demand diagnostics is useful primarily as a  troubleshooting tool to verify hardware functions when an administrator suspects  a hardware fault.

On-demand diagnostics does not cause the faulty hardware to reset or power down the Cisco Catalyst 6500 series. Syslog messages warn about the faulty hardware, and the administrator needs to check the  diagnostics results to see if the tests pass or fail, and then take appropriate action. As an example, this command triggers two on-demand module memory tests (test number 12) on a module in slot 2. If the first memory test fails, no further testing is performed, as shown:

Router#diagnostic ondemand iterations 2
Router#diagnostic ondemand action-on-failure stop
Router#diagnostic start module 2 test ?
Router#diagnostic start module 2 test 12

To display the test suite and the monitoring interval and test attributes for  the module, issue the show diagnostic content command. This example shows how to display the test suite and the monitoring interval and test attributes:

Router# show diagnostic content module       1

Diagnostic Tests List for Module 1:

Module 1:

Diagnostics test suite attributes:

M/C/* - Minimal level test / Complete       level test / Not applicable

B/* - Bypass bootup test / Not       applicable

P/* - Per port test / Not applicable

D/N - Disruptive test / Non-disruptive       test

S/* - Only applicable to standby unit /       Not applicable

X/* - Not a health monitoring test /       Not applicable

F/* - Fixed monitoring interval test /       Not applicable

E/* - Always enabled monitoring test /       Not applicable

A/I - Monitoring is active / Monitoring       is inactive

Testing Interval

ID Test Name Attributes (day       hh:mm:ss.ms)

=== ==================================       ========== =================

1) TestDummy1 ---------------------->       M**D****A 000 00:01:00.000

2) TestDummy2 ---------------------->       M**D**FEA 000 00:02:30.000

3) TestGBICIntegrity --------------->       *BPD****I not configured

4) TestActiveToStandbyLoopback ----->       M*PDS***I not configured

5) TestLoopback -------------------->       M*PD****I not configured

6) TestNewLearn -------------------->       M**N****I not configured

7) TestIndexLearn ------------------>       M**N****I not configured

8) TestConditionalLearn ------------>       M**N****I not configured

9) TestBadBpdu --------------------->       M**D****I not configured

10) TestCapture --------------------->       M**D****I not configured

11) TestProtocolMatch --------------->       M**D****I not configured

12) TestChannel --------------------->       M**D****I not configured

13) TestDontShortcut ---------------->       M**N****I not configured

14) TestL3Capture2 ------------------>       M**N****I not configured

15) TestL3VlanMet ------------------->       M**N****I not configured

16) TestIngressSpan ----------------->       M**N****I not configured

17) TestEgressSpan ------------------>       M**N****I not configured

18) TestAclPermit ------------------->       M**N****I not configured

19) TestAclDeny --------------------->       M**D****I not configured

20) TestNetflowInlineRewrite -------->       C*PD****I not configured

Router#

This example shows how to start a diagnostic test on a specific module:

Router# diagnostic start module 1 test       5

Module 1:Running test(s) 5 may disrupt       normal system operation

Do you want to run disruptive tests? [no]yes

00:48:14:Running OnDemand Diagnostics       [Iteration #1] ...

00:48:14:%DIAG-SP-6-TEST_RUNNING:Module       1:Running TestNewLearn{ID=5} ...

00:48:14:%DIAG-SP-6-TEST_OK:Module       1:TestNewLearn{ID=5} has completed successfully

00:48:14:Running OnDemand Diagnostics       [Iteration #2] ...

00:48:14:%DIAG-SP-6-TEST_RUNNING:Module       1:Running TestNewLearn{ID=5} ...

00:48:14:%DIAG-SP-6-TEST_OK:Module       1:TestNewLearn{ID=5} has completed successfully

Router#

This example shows how to display the online diagnostic results for a module:

Router# show diagnostic result module 5

Current bootup diagnostic level:minimal

Module 5:

Overall Diagnostic Result for Module 5       ASS

Diagnostic level at card bootup:minimal

Test results:(. = Pass, F = Fail, U =       Untested)

1) TestScratchRegister -------------> .

2) TestSPRPInbandPing --------------> .

3) TestGBICIntegrity:

Port 1 2

----------

U U

4) TestActiveToStandbyLoopback:

Port 1 2

----------

U U

5) TestLoopback:

Port 1 2

----------

. .

6) TestNewLearn --------------------> .

7) TestIndexLearn ------------------> .

8) TestDontLearn -------------------> .

9) TestConditionalLearn ------------> .

10) TestBadBpdu --------------------->       .

11) TestTrap ------------------------>       .

12) TestMatch ----------------------->       .

13) TestCapture --------------------->       .

14) TestProtocolMatch --------------->       .

15) TestChannel --------------------->       .

16) TestIPv4FibShortcut ------------->       .

17) TestL3Capture2 ------------------>       .

18) TestL3VlanMet ------------------->       .

19) TestIngressSpan ----------------->       .

20) TestEgressSpan ------------------>       .

21) TestIPv6FibShortcut ------------->       .

22) TestMPLSFibShortcut ------------->       .

23) TestNATFibShortcut -------------->       .

24) TestAclPermit ------------------->       .

25) TestAclDeny --------------------->       .

26) TestQoSTcam --------------------->       .

27) TestNetflowInlineRewrite:

Port 1 2

----------

U U

28) TestFabricSnakeForward ---------->       .

29) TestFabricSnakeBackward --------->       .

30) TestFibTcam - RESET ------------->       U

Router#

This example shows how to display the       detailed online diagnostic results for a module:

Router# show diagnostic result module 5       detail

Current bootup diagnostic level:minimal

Module 5:

Overall Diagnostic Result for Module 5       ASS

Diagnostic level at card bootup:minimal

Test results:(. = Pass, F = Fail, U =       Untested)

___________________________________________________________________________

1) TestScratchRegister -------------> .

Error code ------------------> 0 (DIAG_SUCCESS)

Total run count -------------> 330

Last test execution time ----> May 12       2003 14:49:36

First test failure time -----> n/a

Last test failure time ------> n/a

Last test pass time ---------> May 12       2003 14:49:36

Total failure count ---------> 0

Consecutive failure count ---> 0

___________________________________________________________________________

2) TestSPRPInbandPing --------------> .

Error code ------------------> 0 (DIAG_SUCCESS)

Total run count -------------> 660

Last test execution time ----> May 12       2003 14:49:38

First test failure time -----> n/a

Last test failure time ------> n/a

Last test pass time ---------> May 12       2003 14:49:38

Total failure count ---------> 0

Consecutive failure count ---> 0

___________________________________________________________________________

3) TestGBICIntegrity:

Port 1 2

----------

U U

Error code ------------------> 0 (DIAG_SUCCESS)

Total run count -------------> 0

Last test execution time ----> n/a

First test failure time -----> n/a

Last test failure time ------> n/a

Last test pass time ---------> n/a

Total failure count ---------> 0

Consecutive failure count ---> 0

________________________________________________________________________

Router#

Scheduled diagnostics tests run at either one specific time or periodically. This is useful when scheduling disruptive tests during maintenance windows. When failures are detected, appropriate syslog  messages are displayed. Diagnostics results can be accessed by issuing the show  diagnostic result command on the switch. Scheduled diagnostics does not cause  the faulty hardware to reset or power down the Cisco Catalyst 6500 series. This CLI schedules a loopback test (test number 1) on a module situated in module 2 every Monday at 3 a.m., as shown:

Router(config)#diagnostic schedule       module 2 test 1 weekly MON 03:00

Once testing is complete, capture the test results. If opening a Technical Support case,  include the test results as a case attachment.

For more information on online diagnostics, refer to Generic Online Diagnostics on the Cisco Catalyst 6500 Series Switch.

For detailed configuration options, refer to Configuring Online Diagnostics.