11-28-2013 01:26 AM
I'm running LMS 4.2, and decided to try an evaluation copy of Prime 2.0 and run them side by side.
Initial installation and discovery completed, I left all settings at default (I didn't set up alerting, email, etc; I wanted to use it as a test, not deploy it in production)
For the last few nights, 2 of my 3560G switches have been rebooting; sometimes just one, sometimes both AND at the same time every night, 10 p.m.. The crashinfo output doesn't show any details, and the Cisco output interpreter didn't help.
Setting the switches to syslog to another device, I discover a message "configured from SNMP by (Prime IP address) at 10:00 on every switch; I haven't set any tasks up on Prime, and I don't understand what this is, but it is far too much of a coincidence to ignore.
Both switches are running an older code,
c3560-ipbase-mz.122-25.SEB4.bin
and prior to this have run for 2 years continuously.
Any ideas, anyone?
SR1-8-3560 uptime is 1 day, 11 hours, 48 minutes
System returned to ROM by error - Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!, PC 0x75707C
System restarted at 22:01:23 GMT Tue Nov 26 2013
System image file is "flash:c3560-ipbase-mz.122-25.SEB4/c3560-ipbase-mz.122-25.SEB4.bin"
SR1-7-3560 uptime is 11 hours, 51 minutes
System returned to ROM by error - Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!, PC 0x75707C
System restarted at 22:01:03 GMT Wed Nov 27 2013
System image file is "flash:c3560-ipbase-mz.122-25.SEB4/c3560-ipbase-mz.122-25.SEB4.bin"
Message was edited by: Gareth Tomlinson
Solved! Go to Solution.
11-28-2013 04:04 PM
Any ideas, anyone?
I'd be thinking about upgrading your IOS. Maybe the IOS of your switch has incompatibility. After all, it is old.
11-28-2013 02:10 AM
Hi Gareth,
Can u please paste the crash info file from switch which reboots.
Regards
Sent from Cisco Technical Support iPhone App
11-28-2013 04:28 PM
SR1-8-3560#more crashinfo_2
Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Tue 30-Aug-05 14:19 by yenanh
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x004A54F4 SRR1 = 0x00029210 SRR2 = 0x004A3F68 SRR3 = 0x00020000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000
CPU Register Context:
Vector = 0x00002000 PC = 0x0075707C MSR = 0x00029210 CR = 0x33000005
LR = 0x00756FA4 CTR = 0x001C1794 XER = 0xC0000050
R0 = 0x00000000 R1 = 0x02F71EF8 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x02F71ED0 R7 = 0x00000000
R8 = 0x00029210 R9 = 0x01240000 R10 = 0x01552868 R11 = 0x00000000
R12 = 0x0000001C R13 = 0x00110000 R14 = 0x00000000 R15 = 0x00000200
R16 = 0x03136940 R17 = 0x01240000 R18 = 0x0315A220 R19 = 0x00000003
R20 = 0x00000001 R21 = 0x00FCB59C R22 = 0x00000000 R23 = 0x0176F7D8
R24 = 0x00000000 R25 = 0x009C902C R26 = 0x00000001 R27 = 0x03048078
R28 = 0x00000001 R29 = 0x0133BBA8 R30 = 0x00000000 R31 = 0x01144EE8
Stack trace:
PC = 0x0075707C, SP = 0x02F71EF8
Frame 00: SP = 0x02F71F08 PC = 0x00756F7C
Frame 01: SP = 0x02F71F38 PC = 0x00BF914C
Frame 02: SP = 0x02F71F68 PC = 0x009C9030
Frame 03: SP = 0x02F72088 PC = 0x009C91D8
Frame 04: SP = 0x02F720C8 PC = 0x009CAB34
Frame 05: SP = 0x02F72200 PC = 0x00833A20
Frame 06: SP = 0x02F72220 PC = 0x00834464
Frame 07: SP = 0x02F72298 PC = 0x0086C494
Frame 08: SP = 0x02F722A0 PC = 0x00758DA8
Frame 09: SP = 0x00000000 PC = 0x00752FEC
SR1-8-3560#more crashinfo_1
Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Tue 30-Aug-05 14:19 by yenanh
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x004A54F4 SRR1 = 0x00029210 SRR2 = 0x004A3F6C SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000
CPU Register Context:
Vector = 0x00002000 PC = 0x0075707C MSR = 0x00029210 CR = 0x33000005
LR = 0x00756FA4 CTR = 0x001C1794 XER = 0xC0000050
R0 = 0x00000000 R1 = 0x03194C18 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x03194BF0 R7 = 0x00000000
R8 = 0x00029210 R9 = 0x01240000 R10 = 0x01552868 R11 = 0x00000000
R12 = 0x0000001C R13 = 0x00110000 R14 = 0x00000000 R15 = 0x00000200
R16 = 0x0314ABF0 R17 = 0x01240000 R18 = 0x034112E0 R19 = 0x00000003
R20 = 0x00000001 R21 = 0x00FCB59C R22 = 0x00000000 R23 = 0x0176F7D8
R24 = 0x00000000 R25 = 0x009C902C R26 = 0x00000001 R27 = 0x030E3280
R28 = 0x00000001 R29 = 0x0133BBA8 R30 = 0x00000000 R31 = 0x01144EE8
Stack trace:
PC = 0x0075707C, SP = 0x03194C18
Frame 00: SP = 0x03194C28 PC = 0x00756F7C
Frame 01: SP = 0x03194C58 PC = 0x00BF914C
Frame 02: SP = 0x03194C88 PC = 0x009C9030
Frame 03: SP = 0x03194DA8 PC = 0x009C91D8
Frame 04: SP = 0x03194DE8 PC = 0x009CAB34
Frame 05: SP = 0x03194F20 PC = 0x00833A20
Frame 06: SP = 0x03194F40 PC = 0x00834464
Frame 07: SP = 0x03194FB8 PC = 0x0086C494
Frame 08: SP = 0x03194FC0 PC = 0x00758DA8
Frame 09: SP = 0x00000000 PC = 0x00752FEC
11-28-2013 10:07 PM
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
HI Gareth,
A Cisco Catalyst 3560 switch that runs Cisco IOS Software Release 12.2(20)SE4 or later reloads or resets with the Debug Exception (Could be NULL pointer dereference) Exception (0x2000)! error message.
This problem occurs when the Powered Device (PD) is detected or classified as an overcurrent class.
This issue is documented in Cisco bug ID CSCsa72400.
As a workaround, do not connect 802.3af non-standard class PDs, or even bad or loopback cables, to the switch because the switch can detect the class incorrectly.
In order to resolve this issue, upgrade to Cisco IOS Software Release 12.2(25)SEA on the switch. Alternatively, upgrade to the latest maintenance release, which can be downloaded from Cisco Downloads..
https://supportforums.cisco.com/docs/DOC-4236
Regards
Dont forget to rate helpful posts.
11-29-2013 12:49 AM
unfortunately it isn't a PoE switch; and it's supplied from a APC UPS and a RPS simultaneously for mains power.
Additionally, why would it happen at exactly the same time every day?
Please believe me, I'm very grateful for all these suggestions but I don't think this is the issue.
Gareth
11-28-2013 03:50 PM
If you aren't able to get any hints from Sandeep's suggestion::
You might go to Administration> Background tasks and look for tasks that run for switches - There are only 4 or 5 or so then got through a process of elimination by disabling certain jobs to try to narrow down what might be causing the crash-
11-28-2013 04:30 PM
I've killed the server completely, I can't afford to have switches rebooting randomly. No problem tonmight, I'll let it go another few nights before I'm confident that it WAS Prime, though.
11-28-2013 04:04 PM
Any ideas, anyone?
I'd be thinking about upgrading your IOS. Maybe the IOS of your switch has incompatibility. After all, it is old.
11-28-2013 04:31 PM
I've upgraded one, the trouble is so many clusters spread across the infrastructure that I have to be careful what switch I can take out of service; Killing Prime temporarily seems to have cured the problem, but I'll confirm that after a few more days
11-28-2013 04:39 PM
I've upgraded one, the trouble is so many clusters spread across the infrastructure that I have to be careful what switch I can take out of service; Killing Prime temporarily seems to have cured the problem, but I'll confirm that after a few more days
Here's my dig of what is going on.
1. 12.2(25)SEB is one of the notorious IOS which a lot of CPU hogs issue.
2. 10:00 pm maybe the start of the LMS 4.2 to start collecting data from your switches, however, you may have also some clients doing high-traffic work at that time as well (like backups). Put those two together, add a dash of CPU hog and you have a fine dish of "crash" the next morning. Served warm.
I'm not sure if the developers of PI 2.0 to have switches running IOS this old but I doubt if they have factored that in into the coding.
11-29-2013 12:52 AM
Hi Leo
No problems last night, so I suspect you are on the money here. Yes, there are backups running at that time across the network, every night.
I am wondering what process runs at 10 p.m on Prime, I haven't been able to identify it yet.
Gareth
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide