Hello,

sukhanand_sarkar · ‎10-31-2015

Hello Everyone,

In Switch 6807 , i am observing some logs kindly find the below log for refrence.

==============================================================

%SCHED-SW1-3-THRASHING: Process thrashing on watched queue 'FM Inband ReplyQ'. -Process= "FM core", ipl= 5, pid= 792
-Traceback= 70F5628z 70F5908z 454E5C0z 528AD94z 5284378z
%SCHED-SW1-3-THRASHING: Process thrashing on watched queue 'FM Inband ReplyQ'. -Process= "FM core", ipl= 5, pid= 792
-Traceback= 70F5628z 70F5908z 454E5C0z 528AD94z 5284378z

==============================================================

so i have checked the cpu utilization and found that utilization is normal but as the log shows that the process belongs to FM core so i check FM core process in sh processes cpu sorted

================================================================

sh processes cpu sorted
CPU utilization for five seconds: 10%/0%; one minute: 12%; five minutes: 11%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

792 158776 6855405 23 0.00% 0.00% 0.00% 0 FM core

================================================================

I tried to search it on cisco and google but i didn't find any useful document.

We have not done any changes in network recently so we are not able to analyse why this logs are coming.

If anyone knows about this kindly help me.

cesarfer · ‎10-31-2015

Hello,

Can you please post the output of the "show version" command from the device affected? I will decode these messages and let you know what it is going on.

Thanks,

Cesar Fernandez J

LAN Switchingn Engineer.

sukhanand_sarkar · ‎11-02-2015

Hi Cesar,

Below ios we running

Description:	UPD IP SRV 2 ADV IP ENCRYPT
Release:	15.1.2-SY4a
Release Date:	28/Dec/2014
File Name:	s2t54-advipservicesk9-mz.SPA.151-2.SY4a.bin

cesarfer · ‎11-03-2015

Hello,

I am sorry but in ordre to decode the messages I will need the whole output of the "show version" command.

Our internal tool requires the whole output not just the IOS file name.

Thanks,

Cesar Fernandez J
LAN Switching Engineer.

sukhanand_sarkar · ‎11-09-2015

Hi ,

Cisco IOS Software, s2t54 Software (s2t54-ADVIPSERVICESK9-M), Version 15.1(2)SY4a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 22-Dec-14 12:24 by prod_rel_team

ROM: System Bootstrap, Version 12.2(50r)SYS3, RELEASE SOFTWARE (fc1)

xxxxx uptime is 10 weeks, 3 days, 2 hours, 32 minutes
Uptime for this control processor is 10 weeks, 3 days, 2 hours, 34 minutes
System returned to ROM by power on
System restarted at 11:44:17 UTC Sat Aug 15 2015
System image file is "bootdisk:s2t54-advipservicesk9-mz.SPA.151-2.SY4a.bin"
Last reload reason: power-on

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco C6807-XL (M8572) processor (revision ) with 1785856K/262144K bytes of memory.
Processor board ID xxxxxxxxx
CPU: MPC8572_E, Version: 2.2, (0x80E80022)
CORE: E500, Version: 3.0, (0x80210030)
CPU:1500MHz, CCB:600MHz, DDR:600MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled

Last reset from power-on
31 Virtual Ethernet interfaces
150 Gigabit Ethernet interfaces
36 Ten Gigabit Ethernet interfaces
2543K bytes of non-volatile configuration memory.

Configuration register is 0x2102

cesarfer · ‎11-15-2015

Hello,

I decoded the messages shown above and I got the following conclusion:

cisco C6807-XL (M8572) processor (revision ) with 1785856 bytes of memory.
Configuration register is2102
-Traceback=
      0x70F5628[process_events_waiting_p+0x330]
      0x70F5908[process_wait_for_event+0x98]
      0x454E5C0[fm_core_process+0xc4]
      0x528AD94[ppc_process_dispatch+0x24]
      0x5284378[process_execute+0x244]

1)	The error message we are seeing indicates that there are a consistent number of tasks
present in one of the queues of the Feature Manager core process.
2)	This means that the tasks are either not being done, or the process is overwhelmed and
this task queue is always full.
3)	The Feature manager core handles a lot of processes.
4)	It can also be a seating issue with the Sup or poor connection to the backplane.
5)	Please fail-over (which may clear the issue by clearing the task queue), then reseat
the now standby Supervisor to be sure the module is seated properly.


Thanks,

Cesar Fernandez J

LAN Switching Engineer.

sukhanand_sarkar · ‎11-16-2015

ok so finally the only solution is to go for failover so can you tell the the exact steps for failover test for this issue.

cesarfer · ‎11-17-2015

The failover will be required if you are using redundant supervisor or if you are working on a VSS (Virtual Switch System) environment. If that is your case then perform the following steps:

1. Apply the following commands:

- enable

- redundancy force-switchover

2. Hard reset of the new standby supervisor (old active).

If you don't have redundant supervisors nor VSS, you will need to perform a hard reset of the supervisor located in the chassis.

Thanks,

Cesar Fernandez J
LAN Switching Engineer.

Lohit Jethwani · ‎11-01-2015

Hi Sukhanand,

I researeched further on your problem, however, who ever needs to go deep, needs the following outputs:

show version

show module

show log ( relavent outputs only )

Any high CPU(including interrupt) seen during the time ?

Analysis:

I understand you are seeing a process thrashing message on your switch.
General info about thrashing messages can be found here:
http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-software-releases-121-mai
nline/12422-scheduler-12422.html#ts2

The error message we are seeing indicates that there are a consistent number of tasks
present in one of the queues of the Feature Manager core process. This mean that the tasks
are either not being done, or the process is overwhelmed and this task queue is always
full. The Feature manager core handles processing for a wide variety of features from WCCP
redirection to compression and programming of ACLs into TCAM.

It can also be a seating issue with the Sup or poor connection to the backplane. I would
recommend failing over (which may clear the issue by clearing the task queue), then
reseating the now standby Supervisor to be sure the module is seated properly.

HTH

Lohit Jethwani

TAC

sukhanand_sarkar · ‎11-02-2015

Hi Lohit,

Below ios we running

Description:	UPD IP SRV 2 ADV IP ENCRYPT
Release:	15.1.2-SY4a
Release Date:	28/Dec/2014
File Name:	s2t54-advipservicesk9-mz.SPA.151-2.SY4a.bin

and for show log the output is same which i have shared earlier.

sukhanand_sarkar · ‎11-14-2015

hi Lohit,

please find the show version output.