01-25-2014 08:11 AM - edited 03-07-2019 05:47 PM
Hi Folks,
We recently had a core switch reboot itself (in face it was constantly rebooting into Rommon and then restarted without me doing anything). I have absolutely no idea what caused the problem. I power cycled the switch and it reloaded fine, config all good etc and has worked for the last few days without issue.
Is there anyway I could try to diagnos what went wrong, I have checked sh process cpu and sh platform health, platform health does show that
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
K2PortMan Review 2.00 3.48 15 11 100 500 2 1 1 21:44
but this seems to go up and down a bit and the above actual is the highest I have seen it. Cannt really run debug as this is our core switch and I don't want to load it in anyway.
Worth raising an TAC with cisco maybe ??? I think they are now EOL.
Any advice apreciated.
C.
01-25-2014 09:07 AM
Hi Chris,
Please check the bootflash for a crash file.
Do a dir bootflash:
In the outputs you will see a crashinfo.txt is the switch crashed due to a fault or a memory issue.
Also issue the command 'show version'
In its output you will see a statement as ' system returned to ROM by power on/pc error
If any error is mentioned here or you have a crash file, you may need to get a root cause.
You may also run a couple of diagnostics on the switch. Below is the link to the doc with all the diagnostic commands:
As far as CPU is concerned, provide the output of:
show proc cpu sort | ex 0.00
show proc cpu history
show plat health
Thanks
Ankur
'Please don't forget to rate the post if it helped you'
01-27-2014 01:11 AM
Hi Ankur,
I found a crashinfo: directory. In there I have two files, I have a data file in there which I have looked at and one line is below.
Fatal K Ž2€2 Error FFE 40000 PPE 0
BPCAT41#sh processes cpu | ex 0.00
CPU utilization for five seconds: 19%/2%; one minute: 18%; five minutes: 17%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
15 975356 2919474 334 0.57% 0.55% 0.53% 0 ARP Input
25 18154196 71334268 254 8.09% 9.75% 9.54% 0 Cat4k Mgmt HiPri
26 8094204 17475051 463 5.57% 4.43% 3.91% 0 Cat4k Mgmt LoPri
32 1263084 9050135 139 0.40% 0.57% 0.54% 0 Spanning Tree
35 101288 2106959 48 0.08% 0.06% 0.06% 0 VLAN Manager
39 2008408 8758191 229 1.31% 0.99% 0.93% 0 IP Input
44 209884 368614 569 0.08% 0.12% 0.12% 0 CEF process
62 920 804 1144 0.08% 0.05% 0.01% 1 Virtual Exec
sh proc cpu history, this one doesnt work, its a 10 year old switch afterall
BPCAT41#sh plat health
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
Lj-poll 1.00 0.00 2 0 100 500 0 0 0 0:14
GalChassisVp-review 3.00 0.07 10 99 100 500 0 0 0 2:53
S2w-JobEventSchedule 10.00 0.84 10 12 100 500 0 0 0 35:58
Stub-JobEventSchedul 10.00 1.59 10 4 100 500 1 1 1 56:15
Pim-review 0.10 0.00 1 0 100 500 0 0 0 0:08
Ebm-host-review 1.00 0.00 8 6 100 500 0 0 0 2:07
Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 0:00
Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:00
Acl-Flattener 1.00 0.00 10 5 100 500 0 0 0 0:00
KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 1:45
KxAclPathMan update 2.00 0.00 10 0 100 500 0 0 0 0:00
KxAclPathMan reprogr 1.00 0.00 2 0 100 500 0 0 0 0:00
TagMan-InformMtegRev 1.00 0.00 5 0 100 500 0 0 0 0:00
TagMan-RecreateMtegR 1.00 0.00 10 0 100 500 0 0 0 0:00
K2CpuMan Review 30.00 5.84 30 30 100 500 5 5 5 172:57
K2AccelPacketMan: Tx 10.00 0.99 20 0 100 500 0 0 0 28:30
K2AccelPacketMan: Au 0.10 0.00 0 0 100 500 0 0 0 0:00
K2AclMan-taggedFlatA 1.00 0.00 10 0 100 500 0 0 0 0:00
K2AclCamMan stale en 1.00 0.00 10 0 100 500 0 0 0 0:00
K2AclCamMan hw stats 3.00 0.76 10 5 100 500 0 0 0 14:20
K2AclCamMan kx stats 1.00 0.00 10 5 100 500 0 0 0 6:46
K2AclPolicerTableMan 1.00 0.00 10 0 100 500 0 0 0 0:09
K2L2 Address Table R 2.00 0.00 10 8 100 500 0 0 0 6:38
K2L2 New Static Addr 2.00 0.00 10 6 100 500 0 0 0 0:00
K2L2 New Multicast A 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 Vlan Table Revi 2.00 0.00 12 8 100 500 0 0 0 0:01
K2 L2 Destination Ca 2.00 0.00 10 0 100 500 0 0 0 0:00
K2PortMan Review 2.00 1.99 15 11 100 500 1 1 1 66:37
Gigaport0 Review 0.40 0.03 4 1 100 500 0 0 0 1:31
Gigaport1 Review 0.40 0.04 4 1 100 500 0 0 0 1:36
Gigaport2 Review 0.40 0.04 4 1 100 500 0 0 0 1:36
Gigaport3 Review 0.40 0.04 4 1 100 500 0 0 0 1:36
Gigaport4 Review 0.40 0.03 4 1 100 500 0 0 0 1:31
Gigaport5 Review 0.40 0.04 4 1 100 500 0 0 0 1:33
Gigaport6 Review 0.40 0.04 4 0 100 500 0 0 0 1:12
Gigaport7 Review 0.40 0.04 4 0 100 500 0 0 0 1:24
Gigaport8 Review 0.40 0.14 4 0 100 500 0 0 0 5:40
Gigaport9 Review 0.40 0.15 4 0 100 500 0 0 0 6:07
Gigaport10 Review 0.40 0.13 4 0 100 500 0 0 0 5:50
Gigaport11 Review 0.40 0.17 4 0 100 500 0 0 0 5:38
Gigaport12 Review 0.40 0.04 4 0 100 500 0 0 0 1:24
Gigaport13 Review 0.40 0.03 4 0 100 500 0 0 0 1:12
Gigaport14 Review 0.40 0.03 4 0 100 500 0 0 0 1:11
Gigaport15 Review 0.40 0.03 4 0 100 500 0 0 0 1:10
Gigaport16 Review 0.40 0.03 4 1 100 500 0 0 0 1:21
Gigaport17 Review 0.40 0.03 4 1 100 500 0 0 0 1:17
Gigaport18 Review 0.40 0.04 4 1 100 500 0 0 0 1:36
Gigaport19 Review 0.40 0.04 4 1 100 500 0 0 0 1:32
Gigaport20 Review 0.40 0.03 4 1 100 500 0 0 0 1:18
Gigaport21 Review 0.40 0.04 4 1 100 500 0 0 0 1:33
Gigaport22 Review 0.40 0.04 4 1 100 500 0 0 0 1:28
Gigaport23 Review 0.40 0.03 4 1 100 500 0 0 0 1:32
Gigaport24 Review 0.40 0.16 4 0 100 500 0 0 0 6:08
Gigaport25 Review 0.40 0.13 4 0 100 500 0 0 0 5:11
Gigaport26 Review 0.40 0.13 4 0 100 500 0 0 0 5:08
Gigaport27 Review 0.40 0.16 4 0 100 500 0 0 0 6:02
Gigaport28 Review 0.40 0.16 4 0 100 500 0 0 0 5:09
Gigaport29 Review 0.40 0.17 4 0 100 500 0 0 0 6:06
Gigaport30 Review 0.40 0.16 4 0 100 500 0 0 0 5:07
Gigaport31 Review 0.40 0.14 4 0 100 500 0 0 0 5:04
K2Fib cam usage revi 2.00 0.00 15 0 100 500 0 0 0 0:05
K2Fib IrmFib Review 2.00 0.00 15 7 100 500 0 0 0 0:00
K2Fib AdjRepop Revie 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib Vrf Unpunt Rev 2.00 0.00 15 0 100 500 0 0 0 0:02
K2Fib Consistency Ch 1.00 0.01 5 3 100 500 0 0 0 3:07
K2FibAdjMan Stats Re 2.00 0.64 10 7 100 500 0 0 0 11:10
K2FibAdjMan Host Mov 2.00 0.00 10 9 100 500 0 0 0 21:46
K2FibAdjMan Adj Chan 2.00 0.00 10 0 100 500 0 0 0 0:00
K2FibMulticast Signa 2.00 0.01 10 2 100 500 0 0 0 1:36
K2FibMulticast Entry 2.00 0.00 10 5 100 500 0 0 0 0:00
K2FibMulticast Irm M 2.00 0.00 10 7 100 500 0 0 0 0:00
K2FibFastDropMan Rev 2.00 0.00 7 0 100 500 0 0 0 0:00
K2FibPbr route map r 2.00 0.57 10 7 100 500 0 0 0 6:47
K2FibPbr flat acl pr 2.00 0.07 10 5 100 500 0 0 0 1:28
K2FibPbr consolidati 2.00 0.00 10 0 100 500 0 0 0 0:09
K2FibPerVlanPuntMan 2.00 0.00 15 1 100 500 0 0 0 0:00
K2FibFlowCache flow 2.00 0.02 10 0 100 500 0 0 0 0:24
K2FibFlowCache flow 2.00 0.00 10 0 100 500 0 0 0 0:00
K2FibFlowCache adj r 2.00 0.04 10 0 100 500 0 0 0 0:33
K2MetStatsMan Review 2.00 0.09 5 2 100 500 0 0 0 11:19
K2FibMulticast MET S 2.00 0.00 10 0 100 500 0 0 0 0:00
K2QosDblMan Rate DBL 2.00 0.02 7 0 100 500 0 0 0 1:02
IrmFibThrottler Thro 2.00 0.01 7 1 100 500 0 0 0 0:29
K2 VlanStatsMan Revi 2.00 0.72 15 4 100 500 0 0 0 28:57
K2 L2 Aging Table Re 2.00 0.06 20 2 100 500 0 0 0 5:05
RkiosPortMan Port Re 2.00 0.12 12 8 100 500 0 0 0 5:28
Rkios Module State R 4.00 0.00 40 11 100 500 0 0 0 0:17
RkiosIpPbr IrmPort R 2.00 0.44 10 6 100 500 0 0 0 4:42
RkiosAclMan Review 3.00 0.01 30 0 100 500 0 0 0 0:29
MatMan Review 0.50 0.00 4 0 100 500 0 0 0 0:00
EthHoleLinecardMan(1 2.00 0.00 10 8 100 500 0 0 0 0:19
EthHoleLinecardMan(2 2.00 0.51 10 10 100 500 0 0 0 17:16
EthHoleLinecardMan(3 2.00 0.29 10 10 100 500 0 0 0 17:15
-------------
%CPU Totals 170.80 18.54
Allocation ceiling Current allocation
------------------ ------------------
kbytes % in use kbytes % in use
Linecard 1's Store 258.00 3% 8.65 100%
Linecard 2's Store 258.00 3% 9.58 100%
Linecard 3's Store 258.00 3% 9.58 100%
Linecard 4's Store 258.00 62% 161.08 100%
Linecard 5's Store 258.00 0% 0.00 0%
Linecard 6's Store 258.00 0% 0.00 0%
TSM objects ------------------ ------------------
RkiosSysPacketBuf 250.00 0% 250.00 0%
PacketBufRaw 20355.00 100% 20355.00 100%
PacketBufRawJumbo 732.81 25% 183.20 100%
Packet 1026.56 0% 1026.56 0%
PacketInfoItem 390.62 0% 1.95 0%
VbufNodes2400 80.75 0% 20.18 0%
VbufNodes1600 55.75 0% 13.93 0%
VbufNodes400 73.00 3% 18.25 15%
VbufNodes64 62.00 0% 31.00 0%
PimPhyports 1023.43 24% 413.46 60%
PimPorts 843.75 31% 394.87 66%
PimModules 202.00 1% 12.62 25%
PimSlots 4.00 2% 0.25 37%
PimChassis 2.25 6% 1.12 12%
EbmVlans 2432.00 1% 59.37 49%
EbmVlanHostEntrys 2176.00 2% 68.00 91%
EbmVlanGroupEntrys 1920.00 0% 60.00 10%
EbmPorts 216.00 30% 67.50 96%
EbmPortHostEntrys 384.00 0% 24.00 0%
EbmIeNodes 540.00 0% 5.27 80%
EbmPortVlanAclFeatur 31.99 0% 0.49 0%
EbmSortedHostTableIt 1.87 0% 1.87 0%
EbmSortedGroupTableI 1.75 0% 1.75 0%
IrmVrfs 45.00 1% 0.70 100%
IrmFibAdjs 768.00 3% 96.00 24%
IrmPortEtherAddrEntr 375.00 0% 210.93 0%
IrmFibEntries 9216.00 0% 360.00 12%
AclL4Op 384.00 0% 384.00 0%
AclL4OpTriplet 256.00 0% 256.00 0%
AclClassifier 768.00 0% 768.00 0%
AclFeature 604.50 0% 604.50 0%
Acl 384.00 0% 384.00 0%
Ace 1280.00 0% 1280.00 0%
AceActionDescStorage 256.00 0% 0.00 0%
AclListNode 256.00 0% 256.00 0%
AceListNode 1280.00 0% 1280.00 0%
AclClassifierActionL 512.00 0% 512.00 0%
AclLayerFeatureListN 512.00 0% 512.00 0%
FlatAcl 512.00 0% 512.00 0%
FlatAce 3840.00 0% 3840.00 0%
FlatAceActionListNod 7680.00 0% 7680.00 0%
FlatAclL4OpSetStorag 1024.00 0% 1024.00 0%
FlatAclCacheNode 1024.00 0% 1024.00 0%
QoS Policers 160.00 0% 40.00 0%
KxAclPath 1024.00 0% 128.00 0%
KxAclPathListNode 256.00 0% 32.00 0%
GalGbicEntrys 10.39 0% 10.39 0%
CommandTables 48.00 20% 9.70 100%
K2FibPbrExpandedFlat 2304.00 0% 144.00 1%
K2FibPbrFlatRouteMap 528.00 0% 528.00 0%
K2TxPacket 384.00 0% 0.75 0%
K2TxPacketInfo 256.00 0% 0.50 0%
MatEntrys 5376.00 0% 168.00 0%
MatEntryTableIterato 1.12 0% 1.12 0%
RkiosAclMan NamedGal 305.85 0% 305.85 0%
Rkios Acl VlanMaps 672.00 0% 672.00 0%
Rkios Acl VlanMapEnt 1093.75 0% 1093.75 0%
Rkios QoS PolicyMaps 322.00 0% 56.00 0%
Rkios QoS ClassMaps 4096.00 0% 4096.00 0%
AclToIosFilterMapLis 384.00 0% 384.00 0%
Rkios QoS Policers 32.00 0% 8.00 0%
Event Nodes 84.00 0% 80.00 0%
Event Nodes 84.00 3% 80.00 3%
KxAclTagPairNode 2304.00 0% 2304.00 0%
KxAclMappingTableEnt 96.02 0% 96.02 0%
KxAclMappingTableEnt 64.01 0% 64.01 0%
KxAclTaggedFlatAcl 768.00 0% 768.00 0%
RkisoIpPbrRouteMaps 566.40 0% 7.25 1%
IrmFlows 256.00 0% 64.00 0%
------------------ ------------------
TSM totals 85257.63 25% 55136.21 38%
I couldnt find anything odd in the above.
I havent run any of the diags yet, will they put the device under excessive load, it is our core network switch.
many thanks,
Chris
01-27-2014 02:19 AM
Hi Ankur,
I have looked up the error,
Fatal K Ž2€2 Error FFE 40000 PPE 0
Fatal K2 Error FFE 40000 PPE 0
and this is refered to in field notes as below, however our IOS version is 12.1(19)EW not the one referred to in the field notes.
http://www.cisco.com/en/US/ts/fn/200/fn20248.html
any opinions welcome.
Chris.
01-27-2014 04:39 AM
Found the problem, know bug in the Rom release. CSCed25996
01-27-2014 04:50 AM
Chris,
There's no issue with the CPU.
However as far as the bug is concerned did you match your switch's rommon version, with the affected versions mentioned on the bug?
And can you issue the command "Show platform crashdump" on the switch and provide the outputs?
Thanks
Ankur
"Please rate the post if found useful"
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide