03-28-2015 11:15 AM - edited 03-07-2019 11:17 PM
We have a Cisco 4500 switches and some of them is being reloaded without a known reason, Please check the log output and let me know what should be the cause for that ?
Note that we have Cisco LMS managing the network switches.
***********************************
SW#sh ver
Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-LANBASEK9-M), Version 15.1(2)SG5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 03-Nov-14 09:04 by prod_rel_team
ROM: 12.2(44r)SG5
SW uptime is 1 hour, 42 minutes
System returned to ROM by reload
System restarted at 16:43:51 GMT Sat Mar 28 2015
System image file is "bootflash:cat4500e-lanbasek9-mz.151-2.SG5.bin"
Darkside Revision 4, Nexu Revision 9, Fortooine Revision 1.40
Last reload reason: reload
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
cisco WS-C4503-E (MPC8548) processor (revision 8) with 524288K bytes of memory.
Processor board ID SPE135100L1
MPC8548 CPU at 1GHz, Supervisor 6L-E
Last reset from Reload
2 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.
Configuration register is 0x2101
**********************************************************
SW#sh platform crashdump
========= Start of Crashinfo Collection ======================
Current Time: 3/28/2015 19:7:56
Last Power Failure: 01/21/2010 08:23:28
Last Reload Status: 00000000
Last Software Reset State: 00000201
========= End of Crashinfo Collection ======================
******************************************************************************************
SW#sh environment status
Power Fan Inline
Supply Model No Type Status Sensor Status
------ ---------------- --------- ----------- ------- -------
PS1 PWR-C45-1300ACV AC 1300W good good good
PS2 none -- -- -- -- --
Power supplies needed by system : 1
Power supplies currently available : 1
Chassis Type : WS-C4503-E
Power consumed by backplane : 0 Watts
Switch Bandwidth Utilization : 0%
Supervisor Led Color : Green
Module 1 Status Led Color : Green
Module 2 Status Led Color : Green PoE Led Color : Green
Fantray : Good
Fantray removal timeout : 100
**********************************************************************
*Mar 28 07:55:39.295: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Mar 28 08:55:41.471: %SYS-6-CLOCKUPDATE: System clock has been updated from 07:55:41 UTC Sat Mar 28 2015 to 08:55:41 GMT Sat Mar 28 2015, configured from console by console.
*Mar 28 08:55:41.471: %SYS-6-CLOCKUPDATE: System clock has been updated from 08:55:41 GMT Sat Mar 28 2015 to 08:55:41 GMT Sat Mar 28 2015, configured from console by console.
*Mar 28 08:55:41.803: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 1 (WS-X45-SUP6L-E S/N: JAE14020672 Hw: 1.0) is online
*Mar 28 08:55:41.803: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 2 (WS-X4648-RJ45V+E S/N: JAE14020HCM Hw: 2.0) is online
*Mar 28 08:55:42.335: %SW_VLAN-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to LG.
*Mar 28 08:55:42.387: %SYS-5-CONFIG_I: Configured from memory by console
*Mar 28 08:55:42.551: %SYS-5-RESTART: System restarted --
Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-LANBASEK9-M), Version 15.1(2)SG5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 03-Nov-14 09:04 by prod_rel_team
*Mar 28 08:55:42.639: %SNMP-5-COLDSTART: SNMP agent on host SW is undergoing a cold start
*Mar 28 08:55:42.699: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 20.199.30.37 port 514 started - CLI initiated
*Mar 28 08:55:45.707: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 72.22.66.1 port 514 started - CLI initiated
*Mar 28 08:55:46.443: Port Te1/1: TwinGig Converter inserted: vendor: CISCO SYSTEMS, p/n: 800-27645-02, s/n: FDO13480NT0
*Mar 28 08:55:46.443: Port Te1/2: TwinGig Converter inserted: vendor: CISCO SYSTEMS, p/n: 800-27645-02, s/n: FDO13480NNM
*Mar 28 08:55:48.739: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 72.25.16.25 port 514 started - CLI initiated
**********************************
03-28-2015 01:07 PM
I ran the show ver and show platform crash through Cisco's output interperter and it showed
INFO: The system has been manually rebooted using "reload" command. This is not a system crash.
I do not see a bug that is affecting the relase 15.1(2SG5).
If you will do a "show tech" and upload it, I'll be glad to look at it.
03-28-2015 01:41 PM
03-28-2015 05:10 PM
Looks like there may be a memory leak, however I'm not seeing an error indicating it reloaded due to a memory error.
Do a show process memory several times within the next day or so and verify if the memory utilization continues to increase.
WARNING: The following processes are currently holding more than 1 MB of memory with 'Freed' memory less that 'Holding' memory: 'GaliosObflLogger' (Holding 2206392 bytes) This is considered to be high. High memory usage can be normal, or can indicate a memory leak. To determine this, it is important to see if memory usage changes over time. INFO: A memory leak occurs when a process requests or allocates memory and then forgets to free (de-allocate) the memory when it is finished with that task. As a result, the memory block is reserved until the router is reloaded. Over time, more and more memory blocks are allocated by that process until there is no free memory available. At this point, the router will issue the 'SYS-2-MALLOCFAIL' error message.
TRY THIS: Analyze the 'show process memory' output for this router over a period of time (for example, every few hours or days depending on whether you have a fast or slow leak). Check to see if memory utilization for the affected process(es) continues to increase and the amount of freed memory remains the same. The rate at which free memory disappears depends on how often the event occurs that leads to the leak. A memory leak is a complex condition sometimes requiring an IOS upgrade to correct. If the above is in fact occurring, and you are uncertain about how to proceed, use the Service Request Tool to contact the Cisco TAC for further assistance.
03-28-2015 10:55 PM
Thanks Cehill,
I will run the command tomorrow during the day when there is a lot of traffic going on the switch and see the results.
What do we need to do to fix such issue if it turns out to be memory leaks ?
03-30-2015 06:05 AM
Hey Mohamed,
Upgrade or downgrade can fix a memory leak. Still not sure if this is a memory leak.
Cisco's preferred ios for this platform(gold star beside it) is
15.0.2-SG9(MD).
Take a look at this Caveat/Bug.
Open Caveats for Cisco IOS Release 15.1(2)SG5
CSCum58148
Switch crashes when you modify policer rates of QoS policy..
workaround:
Remove the policy from the targets to which it is attached, modify the policy, and re-attach.
03-30-2015 03:30 PM
Hi cehill,
I have ran the Show Processes Memory couple of times today and the output seems similar 47% usage .
Regarding the upgrade or downgrade I will consider it but not now as I have opened TAC case with Cisco and waiting for analysis.
03-30-2015 06:20 PM
Ok, thanks for the update.
03-31-2015 05:12 AM
Hi,
I have been informed by Cisco that they have a hardware bug CSCtf85481
the Sup Engines will be replaced..all the switches that fails under that hardware revision no. !!
Thanks for your help so far
03-31-2015 05:35 AM
Your welcome.
Thanks for the update.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide