cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6962
Views
10
Helpful
7
Replies

%FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level

Thomas Yarger
Level 1
Level 1

Experts,

Can somebody elaborate on what this error means? According to Cisco docs, " There are too many messages in the queue between the front-end controller and the  switch software." I have not tried rebooting. Wanted to get clarification before doing so.

What kind of messages?

Could this affect PoE operation?

I'm receiving this error on a C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE3, RELEASE SOFTWARE (fc1)

Thanks!      

1 Accepted Solution

Accepted Solutions

Hi Thomas,

Usually when ever you will see the RedEarth Rx Mana process consuming high CPU we will see the following error message:

"%FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller TX queue reached watermark level"

This process is used for Punted Packet Handling. It is normally used for Microprocessor communication process i.e. it does periodic polling on the I2C interface connected to Microprocessor Controller Unit. Microprocessor Controller Unit (MCU) and IOS does sync up on various applications data e.g. PoE, Stack Power etc.

So in this situation when the device is reloaded problem gets fixed.

But if you see this message even without the above mentioned process not consuming high CPU or even after reload the error logs keep appearing, then it means the HW on the switch has gone faulty. And needs replacement.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P


View solution in original post

7 Replies 7

Vignesh Rajendran Praveen
Cisco Employee
Cisco Employee

Hi Thomas,

"%FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level"

This message means that there are too many messages in the queue  between the front-end controller and the switch software.

Yes POE operation might get affected because of this situation.


Try reloading the switch and if this does not resolve the issue, then it should because of a bad hardware. Please have the switch replaced.

This conclusion has been arived upon after researching the internal database.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P

Hi Thomas,

Kindly provide me the output of show proc cpu sorted from the device when these error logs are seen.

Thanks & Regards,

Vignesh R P

Hi Vignesh,

Thank-you for your response. Here is the output. A bit lengthy...

CPU utilization for five seconds: 19%/0%; one minute: 19%; five minutes: 19%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

137   740947204 894156643        828 11.29% 11.15% 11.09%   0 Hulc LED Process

106   278807211  42305118       6590  0.69%  0.64%  0.62%   0 hpm counter proc

191   246414958 905776385        272  0.49%  0.48%  0.50%   0 Spanning Tree

183          68       113        601  0.39%  0.07%  0.01%   1 Virtual Exec

205    79500504  42316228       1878  0.29%  0.27%  0.26%   0 PI MATM Aging Pr

146    89486394   8514169      10510  0.19%  0.19%  0.19%   0 HQM Stack Proces

  68    280875241786717368         15  0.09%  0.05%  0.01%   0 RedEarth Rx Mana

156      943154   4292834        219  0.09%  0.00%  0.00%   0 HL2MCM

245     4830122  25644180        188  0.09%  0.01%  0.00%   0 IGMPSN

  86     73987391193364750          6  0.09%  0.01%  0.00%   0 HLFM address lea

  11           6       195         30  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

  10     3294178  17109401        192  0.00%  0.04%  0.02%   0 ARP Input

   7           0         1          0  0.00%  0.00%  0.00%   0 License Client N

   9           0         1          0  0.00%  0.00%  0.00%   0 Image License br

  12           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit

   8    84051588    709932     118395  0.00%  0.15%  0.15%   0 Licensing Auto U

  17           0         1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager

   5        2587      8827        293  0.00%  0.00%  0.00%   0 Pool Manager

  13           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager

   6           0         2          0  0.00%  0.00%  0.00%   0 Timers

  21           0         1          0  0.00%  0.00%  0.00%   0 IPC Seat Manager

  14          58        46       1260  0.00%  0.00%  0.00%   0 Entity MIB API

  15           0         1          0  0.00%  0.00%  0.00%   0 IFS Agent Manage

  16       11749    709937         16  0.00%  0.00%  0.00%   0 IPC Dynamic Cach

  25           0         1          0  0.00%  0.00%  0.00%   0 License IPC stat

   4   331398921  18495757      17917  0.00%  0.75%  0.76%   0 Check heaps

   3        2002    367523          5  0.00%  0.00%  0.00%   0 DHCPD Timer

  28           0         2          0  0.00%  0.00%  0.00%   0 SMART

  29           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client

  30           0         1          0  0.00%  0.00%  0.00%   0 ARP Snoop

   2      214400   8520845         25  0.00%  0.00%  0.00%   0 Load Meter

  32           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd

  33     9341589   7805325       1196  0.00%  0.01%  0.00%   0 Net Background

  18     2071472  42312441         48  0.00%  0.00%  0.00%   0 IPC Periodic Tim

   1           0        18          0  0.00%  0.00%  0.00%   0 Chunk Manager

  36     2756103  42312273         65  0.00%  0.00%  0.00%   0 TTY Background

  19           0         1          0  0.00%  0.00%  0.00%   0 IPC Managed Time

  20     2027753  42312327         47  0.00%  0.00%  0.00%   0 IPC Deferred Por

  22           0         1          0  0.00%  0.00%  0.00%   0 IPC Session Serv

  40         300     55068          5  0.00%  0.00%  0.00%   0 IF-MGR event pro

  41           0         1          0  0.00%  0.00%  0.00%   0 Inode Table Dest

  42           0         1          0  0.00%  0.00%  0.00%   0 AggMgr Process

  23          57         4      14250  0.00%  0.00%  0.00%   0 PrstVbl

  44      437105  10278512         42  0.00%  0.00%  0.00%   0 HC Counter Timer

  45           0         1          0  0.00%  0.00%  0.00%   0 SFF8472

  24           0         1          0  0.00%  0.00%  0.00%   0 Crash writer

  47           0       217          0  0.00%  0.00%  0.00%   0 EEM ED MAT

  26           0         1          0  0.00%  0.00%  0.00%   0 License IPC serv

  49       60103   1487232         40  0.00%  0.00%  0.00%   0 Net Input

  50     1429560   8515214        167  0.00%  0.00%  0.00%   0 Compute load avg

  27     1980205  42312230         46  0.00%  0.00%  0.00%   0 GraphIt

  52      736399 810526200          0  0.00%  0.00%  0.00%   0 DownWhenLooped

  31     2791845  42312396         65  0.00%  0.00%  0.00%   0 Dynamic ARP Insp

  54           0         1          0  0.00%  0.00%  0.00%   0 HRPC lpip reques

  55           0         2          0  0.00%  0.00%  0.00%   0 HLPIP Sync Proce

  56           0         1          0  0.00%  0.00%  0.00%   0 HRPC hnetwpol re

  57           0         1          0  0.00%  0.00%  0.00%   0 HRPC EnergyWise

  58           0         1          0  0.00%  0.00%  0.00%   0 HRPC actual powe

  59      375357 168960434          2  0.00%  0.00%  0.00%   0 HUSB Console

  60           0         1          0  0.00%  0.00%  0.00%   0 HULC QM Tcam Mem

  34           0         3          0  0.00%  0.00%  0.00%   0 IDB Work

  62           0         3          0  0.00%  0.00%  0.00%   0 Stack DI Update

  63          10        16        625  0.00%  0.00%  0.00%   0 SpanTree Flush

  64           0         1          0  0.00%  0.00%  0.00%   0 HRPC asic-stats

  65       61458  14961862          4  0.00%  0.00%  0.00%   0 Adjust Regions

  66      210762  84599462          2  0.00%  0.00%  0.00%   0 FlexStack Hotswa

  67    29134458 315527863         92  0.00%  0.01%  0.00%   0 RedEarth Tx Mana

  35        7088    142207         49  0.00%  0.00%  0.00%   0 Logger

  37    25463031  42314900        601  0.00%  0.00%  0.00%   0 Per-Second Jobs

  70           0         1          0  0.00%  0.00%  0.00%   0 HRPC hsm request

  71           0         8          0  0.00%  0.00%  0.00%   0 Stack Mgr

  72         183         9      20333  0.00%  0.00%  0.00%   0 Stack Mgr Notifi

  38    14549028    719262      20227  0.00%  0.02%  0.00%   0 Per-minute Jobs

  74      122918   8630318         14  0.00%  0.00%  0.00%   0 hrpc -> request

  39          11         5       2200  0.00%  0.00%  0.00%   0 IF-MGR control p

  76           0         1          0  0.00%  0.00%  0.00%   0 HRPC hcomp reque

  77      467922 208496012          2  0.00%  0.00%  0.00%   0 ppc405ex_emac_pr

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  43          25      6411          3  0.00%  0.00%  0.00%   0 Transport Port A

  79           0         3          0  0.00%  0.00%  0.00%   0 HRPC hdm non blo

  80           0         2          0  0.00%  0.00%  0.00%   0 HRPC hdm blockin

  81      114181   8515129         13  0.00%  0.00%  0.00%   0 HIPC bkgrd proce

  82           0         1          0  0.00%  0.00%  0.00%   0 RTTYS Process

  83       12777    207688         61  0.00%  0.00%  0.00%   0 Hulc Port-Securi

  84           0         1          0  0.00%  0.00%  0.00%   0 HRPC hpsecure re

  85           0         1          0  0.00%  0.00%  0.00%   0 HRPC hlfm reques

  46          11       109        100  0.00%  0.00%  0.00%   0 EEM ED Identity

  87    13810548  42313057        326  0.00%  0.00%  0.00%   0 HLFM aging proce

  88     39525361202755608          3  0.00%  0.00%  0.00%   0 HLFM address ret

  89           0         1          0  0.00%  0.00%  0.00%   0 HRPC hrcmd reque

  90       55276    386867        142  0.00%  0.00%  0.00%   0 HRPC hulc misc r

  48     1150294   7875638        146  0.00%  0.00%  0.00%   0 EEM ED ND

  92      112435  14198626          7  0.00%  0.00%  0.00%   0 HVLAN main bkgrd

  93           0         2          0  0.00%  0.00%  0.00%   0 HVLAN Mapped Vla

  94           0         2          0  0.00%  0.00%  0.00%   0 Vlan shutdown Pr

  51           5         8        625  0.00%  0.00%  0.00%   0 USB Startup

  96           0         7          0  0.00%  0.00%  0.00%   0 HULC VLAN REF Ba

  97           0         1          0  0.00%  0.00%  0.00%   0 HRPC ilp request

  98     9477986  42312941        223  0.00%  0.00%  0.00%   0 Hulc ILP Alchemy

  99           0         1          0  0.00%  0.00%  0.00%   0 HULC PM Vector P

100           0         1          0  0.00%  0.00%  0.00%   0 HPM Msg Retry Pr

101       11262    709939         15  0.00%  0.00%  0.00%   0 DHCPD Database

  53           0         1          0  0.00%  0.00%  0.00%   0 HULC ACL Tcam Me

103        4896    145542         33  0.00%  0.00%  0.00%   0 HPM Stack Sync P

104           0         1          0  0.00%  0.00%  0.00%   0 HRPC pm request

105           0         3          0  0.00%  0.00%  0.00%   0 HPM if_num mappi

  61     36207251566306620          2  0.00%  0.00%  0.00%   0 Draught link sta

107           0         1          0  0.00%  0.00%  0.00%   0 HRPC pm-counters

108           0         1          0  0.00%  0.00%  0.00%   0 hpm vp events ca

109           0         1          0  0.00%  0.00%  0.00%   0 HRPC hcmp reques

110          96     20323          4  0.00%  0.00%  0.00%   0 HCEF ADJ Refresh

111           0         1          0  0.00%  0.00%  0.00%   0 HACL Queue Proce

112           0         1          0  0.00%  0.00%  0.00%   0 HRPC acl request

113           5        88         56  0.00%  0.00%  0.00%   0 HACL Acl Manager

114           0         1          0  0.00%  0.00%  0.00%   0 HRPC aim request

115           0         1          0  0.00%  0.00%  0.00%   0 HRPC backup inte

116         873    142271          6  0.00%  0.00%  0.00%   0 OBFL TEMP obfl0

117           0         1          0  0.00%  0.00%  0.00%   0 HRPC cdp request

118           0         1          0  0.00%  0.00%  0.00%   0 HULC CISP Proces

119           0         1          0  0.00%  0.00%  0.00%   0 HRPC dot1x reque

120           0         1          0  0.00%  0.00%  0.00%   0 Dot1X Msg Retry

121           0         3          0  0.00%  0.00%  0.00%   0 HULC DOT1X Proce

  69      234758   8566631         27  0.00%  0.00%  0.00%   0 HULC Thermal Pro

123           0         1          0  0.00%  0.00%  0.00%   0 HRPC system mtu

124           0         1          0  0.00%  0.00%  0.00%   0 HRPC rep request

125           0         3          0  0.00%  0.00%  0.00%   0 REP Helper Proc

126           0         1          0  0.00%  0.00%  0.00%   0 HRPC sdm request

127           0         1          0  0.00%  0.00%  0.00%   0 SMI MSG Retry Pr

128           0         1          0  0.00%  0.00%  0.00%   0 HRPC Smart Insta

129     3034254 208495995         14  0.00%  0.00%  0.00%   0 Hulc Storm Contr

130           0         2          0  0.00%  0.00%  0.00%   0 HSTP Sync Proces

131           0         1          0  0.00%  0.00%  0.00%   0 HRPC stp_cli req

132           0         1          0  0.00%  0.00%  0.00%   0 HRPC stp_state_s

133           0         2          0  0.00%  0.00%  0.00%   0 S/W Bridge Proce

134           0         1          0  0.00%  0.00%  0.00%   0 HRPC hudld reque

135           0         1          0  0.00%  0.00%  0.00%   0 HRPC vqpc reques

136           0         1          0  0.00%  0.00%  0.00%   0 HRPC hled reques

  73     5786350  42312728        136  0.00%  0.00%  0.00%   0 hrpc -> response

138      899566  31233243         28  0.00%  0.00%  0.00%   0 HL3U bkgrd proce

139           0         1          0  0.00%  0.00%  0.00%   0 HRPC hl3u reques

140           0         1          0  0.00%  0.00%  0.00%   0 HRPC obfl reques

142           0         1          0  0.00%  0.00%  0.00%   0 HRPC dtp request

143           0         1          0  0.00%  0.00%  0.00%   0 HRPC show_forwar

144           0         1          0  0.00%  0.00%  0.00%   0 HRPC snmp reques

145           0         1          0  0.00%  0.00%  0.00%   0 HULC SNMP Proces

  75      621449   8678960         71  0.00%  0.00%  0.00%   0 hrpc <- response

147     7229405  17028423        424  0.00%  0.01%  0.00%   0 HRPC qos request

148           0         1          0  0.00%  0.00%  0.00%   0 HRPC span reques

149           0         1          0  0.00%  0.00%  0.00%   0 HRPC system post

150           0         1          0  0.00%  0.00%  0.00%   0 Hulc Reload Mana

  78           0         6          0  0.00%  0.00%  0.00%   0 HULC Device Mana

152         671    237123          2  0.00%  0.00%  0.00%   0 HCMP sync proces

153           0         2          0  0.00%  0.00%  0.00%   0 image mgr

154           0         1          0  0.00%  0.00%  0.00%   0 HRFS OIR Proc

155     1088810  14198614         76  0.00%  0.00%  0.00%   0 Power RPS Proces

  91           0         5          0  0.00%  0.00%  0.00%   0 HRPC emac reques

157          10         4       2500  0.00%  0.00%  0.00%   0 HL2MCM

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

158         521      2077        250  0.00%  0.00%  0.00%   0 AAA Server

159           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc

160           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr

161           0         1          0  0.00%  0.00%  0.00%   0 AAA System Acct

162           0         1          0  0.00%  0.00%  0.00%   0 AUTH POLICY Fram

163        4667    104757         44  0.00%  0.00%  0.00%   0 Auth Manager

164           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B

165         387    142272          2  0.00%  0.00%  0.00%   0 IP Admin SM Proc

166    14293061  13277836       1076  0.00%  0.01%  0.00%   0 CDP Protocol

167       72945     52523       1388  0.00%  0.00%  0.00%   0 SpanTree Helper

168           5        67         74  0.00%  0.00%  0.00%   0 ENABLE AAA

169           0         1          0  0.00%  0.00%  0.00%   0 CMD HANDLER

170           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R

171    12615868  54865682        229  0.00%  0.01%  0.00%   0 DHCP Snooping

172           0         2          0  0.00%  0.00%  0.00%   0 Dot1x Mgr Proces

173           0         1          0  0.00%  0.00%  0.00%   0 EAP Framework

174           0         1          0  0.00%  0.00%  0.00%   0 EAP Test

175           0         3          0  0.00%  0.00%  0.00%   0 CEF switching ba

176     3725315  29677283        125  0.00%  0.01%  0.00%   0 IP Input

177           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl

178      287120  82801906          3  0.00%  0.00%  0.00%   0 IP ARP Track

179           0         3          0  0.00%  0.00%  0.00%   0 ADJ resolve proc

180           0         2          0  0.00%  0.00%  0.00%   0 ADJ NSF process

181           0         1          0  0.00%  0.00%  0.00%   0 IPv6 ping proces

182           0         1          0  0.00%  0.00%  0.00%   0 MAB Framework

  95           6        39        153  0.00%  0.00%  0.00%   0 HRPC vlan reques

184           0         2          0  0.00%  0.00%  0.00%   0 REP Topology cha

185           0         1          0  0.00%  0.00%  0.00%   0 SMI Director DB

186           0         1          0  0.00%  0.00%  0.00%   0 SMI CDP Update H

187           0         1          0  0.00%  0.00%  0.00%   0 SMI Backup Proce

188           0         2          0  0.00%  0.00%  0.00%   0 SMI IBC server p

189           0         1          0  0.00%  0.00%  0.00%   0 SMI IBC client p

190           0         2          0  0.00%  0.00%  0.00%   0 SMI IBC Download

102    23958337 425994344         56  0.00%  0.03%  0.04%   0 hpm main process

192        1338    711339          1  0.00%  0.00%  0.00%   0 Spanning Tree St

193       15412    104779        147  0.00%  0.00%  0.00%   0 802.1x switch

194       16437    709935         23  0.00%  0.00%  0.00%   0 802.1x MDA Aging

195           0         1          0  0.00%  0.00%  0.00%   0 802.1x Webauth F

196           0         2          0  0.00%  0.00%  0.00%   0 DTP Protocol

197           0         1          0  0.00%  0.00%  0.00%   0 HRPC IPv6 Host r

198           0         2          0  0.00%  0.00%  0.00%   0 IPv6 Platform Ho

199           0         1          0  0.00%  0.00%  0.00%   0 HRPC dai request

200           0         1          0  0.00%  0.00%  0.00%   0 HULC DAI Process

201           0         1          0  0.00%  0.00%  0.00%   0 HRPC power down

202           0         1          0  0.00%  0.00%  0.00%   0 HRPC ip device t

203           0         1          0  0.00%  0.00%  0.00%   0 HRPC ip source g

204           0         1          0  0.00%  0.00%  0.00%   0 HULC IP Source g

122           0         1          0  0.00%  0.00%  0.00%   0 HRPC lldp reques

206      300749  42682034          7  0.00%  0.00%  0.00%   0 UDLD

207        3524   1422736          2  0.00%  0.00%  0.00%   0 Port-Security

208           0         2          0  0.00%  0.00%  0.00%   0 Switch Backup In

209           0         2          0  0.00%  0.00%  0.00%   0 IP Host Track Pr

210           0         1          0  0.00%  0.00%  0.00%   0 Link State Group

211       10144    709935         14  0.00%  0.00%  0.00%   0 MMN bkgrd proces

212       21282   4259587          4  0.00%  0.00%  0.00%   0 Ethchnl

213       13719     52480        261  0.00%  0.00%  0.00%   0 VMATM Callback

214       59307    851760         69  0.00%  0.00%  0.00%   0 CEF background p

215           0         1          0  0.00%  0.00%  0.00%   0 IP IRDP

216           0         1          0  0.00%  0.00%  0.00%   0 CEF RF HULC Conv

217           0         2          0  0.00%  0.00%  0.00%   0 XDR mcast

218       25728   4259588          6  0.00%  0.00%  0.00%   0 IP ACL XDR LC Ba

219           0         1          0  0.00%  0.00%  0.00%   0 IPC LC Message H

220           0         1          0  0.00%  0.00%  0.00%   0 XDR RP Ping Back

221        1056    355669          2  0.00%  0.00%  0.00%   0 XDR RP backgroun

222           0         1          0  0.00%  0.00%  0.00%   0 XDR RP Test Back

223           0         1          0  0.00%  0.00%  0.00%   0 Critical Auth

224        2153     99951         21  0.00%  0.00%  0.00%   0 TCP Timer

225         591      1004        588  0.00%  0.00%  0.00%   0 TCP Protocols

226      791873  42305124         18  0.00%  0.00%  0.00%   0 Socket Timers

227        3975    145328         27  0.00%  0.00%  0.00%   0 HTTP CORE

228        1165    533513          2  0.00%  0.00%  0.00%   0 Cluster L2

229       36026   4259590          8  0.00%  0.00%  0.00%   0 Cluster RARP

230       31581   5334076          5  0.00%  0.00%  0.00%   0 Cluster Base

231    14155422  84610254        167  0.00%  0.03%  0.00%   0 DHCPD Receive

232           0         2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant

233           0         2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant

234           0         2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant

235           0         1          0  0.00%  0.00%  0.00%   0 RARP Input

236           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

237           0         1          0  0.00%  0.00%  0.00%   0 IP RIB Update

238           5         1       5000  0.00%  0.00%  0.00%   0 LICENSE AGENT

239           0         2          0  0.00%  0.00%  0.00%   0 REP LSL Proc

240           0         2          0  0.00%  0.00%  0.00%   0 REP BPA/EPA Proc

241           0         1          0  0.00%  0.00%  0.00%   0 HRPC dhcp snoopi

242    10830484  54517197        198  0.00%  0.00%  0.00%   0 HULC DHCP Snoopi

243      341369   2047630        166  0.00%  0.00%  0.00%   0 IGMPSN L2MCM

244     1845731  16284423        113  0.00%  0.00%  0.00%   0 IGMPSN MRD

151           0         1          0  0.00%  0.00%  0.00%   0 HRPC hrcli-event

246           0         1          0  0.00%  0.00%  0.00%   0 IGMPQR

247           0         2          0  0.00%  0.00%  0.00%   0 L2TRACE SERVER

248    12175482  45633763        266  0.00%  0.00%  0.00%   0 Inline Power

249     5192709  94419728         54  0.00%  0.00%  0.00%   0 Marvell wk-a Pow

250      298456   2047628        145  0.00%  0.00%  0.00%   0 MLDSN L2MCM

251           0         1          0  0.00%  0.00%  0.00%   0 MRD

252           0         1          0  0.00%  0.00%  0.00%   0 MLD_SNOOP

253           0         1          0  0.00%  0.00%  0.00%   0 HRPC hl2mcm igmp

254           0         1          0  0.00%  0.00%  0.00%   0 HRPC hl2mcm mlds

255          41       994         41  0.00%  0.00%  0.00%   0 LOCAL AAA

256           0         1          0  0.00%  0.00%  0.00%   0 EPM MAIN PROCESS

257      378241     13546      27922  0.00%  0.00%  0.00%   0 OBFL UPTIME obfl

258           0         2          0  0.00%  0.00%  0.00%   0 TPLUS

259       37360      2499      14949  0.00%  0.00%  0.00%   0 OBFL MSG obfl0

260           5        12        416  0.00%  0.00%  0.00%   0 crypto engine pr

261          26         4       6500  0.00%  0.00%  0.00%   0 Crypto CA

262           0         1          0  0.00%  0.00%  0.00%   0 Crypto PKI-CRL

263           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL

264           0         1          0  0.00%  0.00%  0.00%   0 encrypt proc

265           0         1          0  0.00%  0.00%  0.00%   0 HRPC x_setup req

266           0         2          0  0.00%  0.00%  0.00%   0 REP Switch Helpe

267           0         1          0  0.00%  0.00%  0.00%   0 Licensing MIB pr

268          15        73        205  0.00%  0.00%  0.00%   0 VTP Trap Process

269         214      1032        207  0.00%  0.00%  0.00%   0 AAA SEND STOP EV

270           0         2          0  0.00%  0.00%  0.00%   0 DHCP Security He

271           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps

272           0         2          0  0.00%  0.00%  0.00%   0 VTPMIB EDIT BUFF

273           0         1          0  0.00%  0.00%  0.00%   0 HCD Process

274           0         1          0  0.00%  0.00%  0.00%   0 HRPC cable diagn

275           0         2          0  0.00%  0.00%  0.00%   0 DiagCard1/-1

276     1008019 182126662          5  0.00%  0.00%  0.00%   0 PM Callback

277     2136538  14240951        150  0.00%  0.00%  0.00%   0 VLAN Manager

278          42         1      42000  0.00%  0.00%  0.00%   0 OBFL ENV obfl0

279       13503   3413132          3  0.00%  0.00%  0.00%   0 dhcp snooping sw

280       43448    709931         61  0.00%  0.00%  0.00%   0 OBFL VOLT obfl0

281           0        14          0  0.00%  0.00%  0.00%   0 EEM Server

282           0         2          0  0.00%  0.00%  0.00%   0 EEM Policy Direc

283           0         1          0  0.00%  0.00%  0.00%   0 tHUB

284       52894     73422        720  0.00%  0.00%  0.00%   0 Syslog

285           0         2          0  0.00%  0.00%  0.00%   0 STP FAST TRANSIT

286           0         2          0  0.00%  0.00%  0.00%   0 CSRT RAPID TRANS

287        5263    142305         36  0.00%  0.00%  0.00%   0 OBFL POE obfl0

288    12378781    734816      16846  0.00%  0.00%  0.10%   0 crypto sw pk pro

289        5289     23707        223  0.00%  0.00%  0.00%   0 SSH Event handle

290           0        16          0  0.00%  0.00%  0.00%   0 SNMP Timers

291    11363542  40177387        282  0.00%  0.00%  0.00%   0 IP SNMP

292     3242930  20101805        161  0.00%  0.00%  0.00%   0 PDU DISPATCHER

293    46561288  20294276       2294  0.00%  0.00%  0.00%   0 SNMP ENGINE

294           0         2          0  0.00%  0.00%  0.00%   0 IP SNMPV6

295           0         1          0  0.00%  0.00%  0.00%   0 SNMP ConfCopyPro

296         487       115       4234  0.00%  0.00%  0.00%   0 SNMP Traps

297     40617701193366205          3  0.00%  0.00%  0.00%   0 RADIUS

298           0        13          0  0.00%  0.00%  0.00%   0 hulc cfg mgr mas

299       14767        80     184587  0.00%  0.00%  0.00%   0 hulc running con

300      277058   2840377         97  0.00%  0.00%  0.00%   0 OBFL I/O Buffer

Hi Thomas,

At the time you collected the "show proc cpu sorted" output where the "%FRNTEND_CTRLR-1-MGR_TXQ_FULL" error los still being seen ?

Thanks & Regards,

Vignesh  R P

Hi Thomas,

Usually when ever you will see the RedEarth Rx Mana process consuming high CPU we will see the following error message:

"%FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller TX queue reached watermark level"

This process is used for Punted Packet Handling. It is normally used for Microprocessor communication process i.e. it does periodic polling on the I2C interface connected to Microprocessor Controller Unit. Microprocessor Controller Unit (MCU) and IOS does sync up on various applications data e.g. PoE, Stack Power etc.

So in this situation when the device is reloaded problem gets fixed.

But if you see this message even without the above mentioned process not consuming high CPU or even after reload the error logs keep appearing, then it means the HW on the switch has gone faulty. And needs replacement.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P


Hello!

 

We're currently running 12.2(55r)SE on a WS-C2960S-48LPS-L and the switch started writing the following messages:

 

Aug 9 07:30:19 xxxxxxxx 56: Aug 9 05:30:32.069: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:35:20 xxxxxxxx 57: Aug 9 05:35:32.910: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:40:21 xxxxxxxx 58: Aug 9 05:40:33.761: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:45:21 xxxxxxxx 59: Aug 9 05:45:33.788: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:50:21 xxxxxxxx 60: Aug 9 05:50:34.545: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:55:22 xxxxxxxx 61: Aug 9 05:55:35.196: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 08:00:22 xxxxxxxx 62: Aug 9 06:00:35.286: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level

...

until Aug 13 11:57:52 - then it was rebooted for changing power cord.

 

about 4hrs before (in between nothing!) also this messages have been received:

 

Aug 9 03:11:48 xxxxxxxx 54: Aug 9 01:12:01.573: %HARDWARE-3-MCU_I2C_READ_ERROR: I2C read failed in MCU and attached devices (e.g. Temperature Sensor) can't be read.
Aug 9 03:11:48 xxxxxxxx 55: Aug 9 01:12:02.574: %HARDWARE-3-MCU_I2C_BAD_DEV: I2C bad device address: 4 (alchemy:0)

 

As a consequent the switch turned of all power feeds to POE devices (12 IP cams class 3 with 15,4W per device)!

 

Some additional informations: this actual switch already has been replaced by RMA about 4 weeks ago because the old one suddenly began to shutdown itself completly. The new one also shows the same symptoms after some days. Powercord has been replaced and is pluged together with an other switch which doesn't show this problem.

The POE devices now have been attached to power injectors to get sure that this is not the trigger.

 

So is this a hardware problem caused by one or more PE devices or the switch itself or is it a software problem indeed?

 

Best regards,

Herbert

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: