Re: Cat 3560 intermittent reboot

Rick Hislop · ‎10-14-2010

Has anyone ever encountered a 3560 intermittently rebooting it

self? I'd appreciate any information on how to track/troubleshoot.

Thanks,

Rick

Leo Laohoo · ‎10-14-2010

Can you console into the switch and get the "crashinfo" file? What IOS are you running?

Rick Hislop · ‎10-15-2010

Here's the output from the show stacks command.

Minimum process stacks:
Free/Size   Name
5112/6000   hulc_flash init
5740/6000   CDP BLOB
8576/9000   IP Background
10364/12000 master cfg mgr init process
5660/6000   vqpc_shim_create_addr_tbl
5648/6000   SPAN Subsystem
5644/6000   bulk config process
5736/6000   hpm bulk_vlan_state process
8732/9000   cdp init process
8688/12000 Init
5492/6000   RADIUS INITCONFIG
5640/6000   Hulc Backup Interface Process
2600/3000   Rom Random Update Process
5484/6000   MDFS LC IPC Init Process
17508/24000 HTTP CP
35184/36000 TCP Command

Interrupt level stacks:
Level    Called Unused/Size Name
4 2930314646   8404/9000 NETWORK INTERFACE CHIP
5           0   9000/9000 SUPERVISOR EXCEPTIONS

Interrupt level stacks:
Level Called Unused/Size Name
6 66617792 8924/9000 NS16550 VECTOR

andtoth · ‎10-15-2010

Hi,

As mentioned, a complete crashinfo file might help finding the root cause. Could you please attach it? You can find it on the flash: of the switch.

Andras

Rick Hislop · ‎10-14-2010

I can try hooking up a console in the morning. The os version is 12.2(35)SE5-IP-BASE

Leo Laohoo · ‎10-17-2010

A crashinfo file would be very useful and that's an OLD IOS.

glen.grant · ‎10-18-2010

Yup the first thing you would want to do is to upgrade the IOS , that is very old and a lot of fixes have been incorporated since then . If the problem continues after that then you are most likely looking at a hardware issue and would have to RMA the box . Have never seen your issue on a 3560 and we have a lot of them installed .

Rick Hislop · ‎10-18-2010

Ok thanks Gents. I'll try a SW upgrade and see if the situation improves.

Cheers,

Rick

Rick Hislop · ‎03-02-2011

Unbeknownst to me until today, the issue has persisted. I managed to catch it while it was down. The status lights o

n the left were all on except the very top one which would blink intermittently. None of the port lights were on

. I didn't get a chance to plug in a console to see what was going on but by the time I got a machine to use to console, the machine had come back up.

Quite a frustrating issue and very disruptive for the users connected to it. There's an older 2950 connected to it and it's fine. No reboots or connectivity issues - other than when this one bounces.

Any other ideas on how to get to the bottom of this?

Thanks,

Rick

Leo Laohoo · ‎03-02-2011

Would be nice if you can provide us with more information. A crashinfo file(s) is very useful.

Have you tried upgrading the IOS?

There's an older 2950 connected to it and it's fine.

That's because the IOS for the 2950 is very different to the 3560.

Rick Hislop · ‎03-02-2011

Yes I understand there's a huge difference between a 2950 and 3560. Sorry I was being facetious.

Below is the info from show stacks

Minimum process stacks:
Free/Size   Name
4904/6000   hulc_flash init
5720/6000   CDP BLOB
5056/6000   MDFS LC IPC Init Process
8284/9000   IP Background
8500/9000   EEM ED RF
4168/6000   EEM Shell Director
5372/6000   vqpc_shim_create_addr_tbl
5360/6000   SPAN Subsystem
9848/12000 master cfg mgr init process
5508/6000   bulk config process
5516/6000   MDFS Reload
5336/6000   REP LSL Hello PP Process
58800/60000 EEM Auto Registration Proc
5328/6000   SASL MAIN
5524/6000   hpm bulk_vlan_state process
7888/12000 Init
8516/9000   cdp init process
5184/6000   RADIUS INITCONFIG
5496/6000   Hulc Backup Interface Process
5476/6000   URPF stats
2532/3000   Rom Random Update Process
Minimum process stacks:
Free/Size   Name
15968/24000 HTTP CP
35068/36000 TCP Command

Interrupt level stacks:
Level    Called Unused/Size Name
4   102474702   8360/9000 NETWORK INTERFACE CHIP
5           0   9000/9000 SUPERVISOR EXCEPTIONS
6     2798718   8928/9000 NS16550 VECTOR

Leo Laohoo · ‎03-02-2011

Honestly, the information doesn't give me anything constructive.

Can you post the following:

1. sh version

2. dir

Have you tried upgrading the IOS? Judging that this last question has been asked numerous times, I am guessing that you're still running the old 12.2(35)SE. Am I correct?

Rick Hislop · ‎03-03-2011

I did upgrade the OS per recommendations.

Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(55)SE, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sat 07-Aug-10 22:26 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02D00000

ROM: Bootstrap program is C3560 boot loader
BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

HO48-2 uptime is 1 hour, 3 minutes
System returned to ROM by power-on
System restarted at 15:29:18 PST Wed Mar 2 2011
System image file is "flash:c3560-ipbasek9-mz.122-55.SE/c3560-ipbasek9-mz.122-55.SE.bin"

cisco WS-C3560G-48PS (PowerPC405) processor (revision F0) with 131072K bytes of memory.
Processor board ID FOC1244W11N
Last reset from power-on
1 Virtual Ethernet interface
52 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:23:AC:DF:85:00
Motherboard assembly number     : 73-10212-04
Power supply part number        : 341-0108-03
Motherboard serial number       : FOC1244028X
Power supply serial number      : DCA1239A8FE
Model revision number           : F0
Motherboard revision number     : C0
Model number                    : WS-C3560G-48PS-S
System serial number            : FOC1244W11N
Top Assembly Part Number        : 800-26845-01
Top Assembly Revision Number    : D0
Version ID                      : V05
CLEI Code Number                : CNMWW00ARC
Hardware Board Revision Number : 0x09

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 52    WS-C3560G-48PS     12.2(55)SE            C3560-IPBASEK9-M

Configuration register is 0xF

Directory of flash:/

    2 -rwx         796   Sep 9 2010 08:22:30 -07:00 vlan.dat
    3 -rwx        3096 Feb 28 1993 16:02:46 -08:00 multiple-fs
    4 -rwx        3160 Feb 25 2011 09:38:50 -08:00 config.text
    5 -rwx        1937 Feb 25 2011 09:38:50 -08:00 private-config.text
    6 drwx         192   Nov 2 2010 17:39:01 -07:00 c3560-ipbasek9-mz.122-55.SE

Leo Laohoo · ‎03-03-2011

Ok. Thanks for this.

Let's observe to see if the 3560 continuously reboots after the IOS upgrade. If you have a spare client/laptop, connect this switch via the console cable. If the switch reboots, it should be captured by the console session for possible troubleshooting.