Solved: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level

Thomas Yarger · ‎08-12-2013

Experts,

Can somebody elaborate on what this error means? According to Cisco docs, " There are too many messages in the queue between the front-end controller and the switch software." I have not tried rebooting. Wanted to get clarification before doing so.

What kind of messages?

Could this affect PoE operation?

I'm receiving this error on a C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE3, RELEASE SOFTWARE (fc1)

Thanks!

Vignesh Rajendran Praveen · ‎08-12-2013

Hi Thomas,

Usually when ever you will see the RedEarth Rx Mana process consuming high CPU we will see the following error message:

"%FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller TX queue reached watermark level"

This process is used for Punted Packet Handling. It is normally used for Microprocessor communication process i.e. it does periodic polling on the I2C interface connected to Microprocessor Controller Unit. Microprocessor Controller Unit (MCU) and IOS does sync up on various applications data e.g. PoE, Stack Power etc.

So in this situation when the device is reloaded problem gets fixed.

But if you see this message even without the above mentioned process not consuming high CPU or even after reload the error logs keep appearing, then it means the HW on the switch has gone faulty. And needs replacement.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P

View solution in original post

Vignesh Rajendran Praveen · ‎08-12-2013

Hi Thomas,

"%FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level"

This message means that there are too many messages in the queue between the front-end controller and the switch software.

Yes POE operation might get affected because of this situation.

Try reloading the switch and if this does not resolve the issue, then it should because of a bad hardware. Please have the switch replaced.

This conclusion has been arived upon after researching the internal database.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P

Vignesh Rajendran Praveen · ‎08-12-2013

Hi Thomas,

Kindly provide me the output of show proc cpu sorted from the device when these error logs are seen.

Thanks & Regards,

Vignesh R P

Thomas Yarger · ‎08-12-2013

Hi Vignesh,

Thank-you for your response. Here is the output. A bit lengthy...

CPU utilization for five seconds: 19%/0%; one minute: 19%; five minutes: 19%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

137 740947204 894156643 828 11.29% 11.15% 11.09% 0 Hulc LED Process

106 278807211 42305118 6590 0.69% 0.64% 0.62% 0 hpm counter proc

191 246414958 905776385 272 0.49% 0.48% 0.50% 0 Spanning Tree

183 68 113 601 0.39% 0.07% 0.01% 1 Virtual Exec

205 79500504 42316228 1878 0.29% 0.27% 0.26% 0 PI MATM Aging Pr

146 89486394 8514169 10510 0.19% 0.19% 0.19% 0 HQM Stack Proces

68 280875241786717368 15 0.09% 0.05% 0.01% 0 RedEarth Rx Mana

156 943154 4292834 219 0.09% 0.00% 0.00% 0 HL2MCM

245 4830122 25644180 188 0.09% 0.01% 0.00% 0 IGMPSN

86 73987391193364750 6 0.09% 0.01% 0.00% 0 HLFM address lea

11 6 195 30 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

10 3294178 17109401 192 0.00% 0.04% 0.02% 0 ARP Input

7 0 1 0 0.00% 0.00% 0.00% 0 License Client N

9 0 1 0 0.00% 0.00% 0.00% 0 Image License br

12 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

8 84051588 709932 118395 0.00% 0.15% 0.15% 0 Licensing Auto U

17 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

5 2587 8827 293 0.00% 0.00% 0.00% 0 Pool Manager

13 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

21 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager

14 58 46 1260 0.00% 0.00% 0.00% 0 Entity MIB API

15 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage

16 11749 709937 16 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

25 0 1 0 0.00% 0.00% 0.00% 0 License IPC stat

4 331398921 18495757 17917 0.00% 0.75% 0.76% 0 Check heaps

3 2002 367523 5 0.00% 0.00% 0.00% 0 DHCPD Timer

28 0 2 0 0.00% 0.00% 0.00% 0 SMART

29 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

30 0 1 0 0.00% 0.00% 0.00% 0 ARP Snoop

2 214400 8520845 25 0.00% 0.00% 0.00% 0 Load Meter

32 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

33 9341589 7805325 1196 0.00% 0.01% 0.00% 0 Net Background

18 2071472 42312441 48 0.00% 0.00% 0.00% 0 IPC Periodic Tim

1 0 18 0 0.00% 0.00% 0.00% 0 Chunk Manager

36 2756103 42312273 65 0.00% 0.00% 0.00% 0 TTY Background

19 0 1 0 0.00% 0.00% 0.00% 0 IPC Managed Time

20 2027753 42312327 47 0.00% 0.00% 0.00% 0 IPC Deferred Por

22 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv

40 300 55068 5 0.00% 0.00% 0.00% 0 IF-MGR event pro

41 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

42 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process

23 57 4 14250 0.00% 0.00% 0.00% 0 PrstVbl

44 437105 10278512 42 0.00% 0.00% 0.00% 0 HC Counter Timer

45 0 1 0 0.00% 0.00% 0.00% 0 SFF8472

24 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

47 0 217 0 0.00% 0.00% 0.00% 0 EEM ED MAT

26 0 1 0 0.00% 0.00% 0.00% 0 License IPC serv

49 60103 1487232 40 0.00% 0.00% 0.00% 0 Net Input

50 1429560 8515214 167 0.00% 0.00% 0.00% 0 Compute load avg

27 1980205 42312230 46 0.00% 0.00% 0.00% 0 GraphIt

52 736399 810526200 0 0.00% 0.00% 0.00% 0 DownWhenLooped

31 2791845 42312396 65 0.00% 0.00% 0.00% 0 Dynamic ARP Insp

54 0 1 0 0.00% 0.00% 0.00% 0 HRPC lpip reques

55 0 2 0 0.00% 0.00% 0.00% 0 HLPIP Sync Proce

56 0 1 0 0.00% 0.00% 0.00% 0 HRPC hnetwpol re

57 0 1 0 0.00% 0.00% 0.00% 0 HRPC EnergyWise

58 0 1 0 0.00% 0.00% 0.00% 0 HRPC actual powe

59 375357 168960434 2 0.00% 0.00% 0.00% 0 HUSB Console

60 0 1 0 0.00% 0.00% 0.00% 0 HULC QM Tcam Mem

34 0 3 0 0.00% 0.00% 0.00% 0 IDB Work

62 0 3 0 0.00% 0.00% 0.00% 0 Stack DI Update

63 10 16 625 0.00% 0.00% 0.00% 0 SpanTree Flush

64 0 1 0 0.00% 0.00% 0.00% 0 HRPC asic-stats

65 61458 14961862 4 0.00% 0.00% 0.00% 0 Adjust Regions

66 210762 84599462 2 0.00% 0.00% 0.00% 0 FlexStack Hotswa

67 29134458 315527863 92 0.00% 0.01% 0.00% 0 RedEarth Tx Mana

35 7088 142207 49 0.00% 0.00% 0.00% 0 Logger

37 25463031 42314900 601 0.00% 0.00% 0.00% 0 Per-Second Jobs

70 0 1 0 0.00% 0.00% 0.00% 0 HRPC hsm request

71 0 8 0 0.00% 0.00% 0.00% 0 Stack Mgr

72 183 9 20333 0.00% 0.00% 0.00% 0 Stack Mgr Notifi

38 14549028 719262 20227 0.00% 0.02% 0.00% 0 Per-minute Jobs

74 122918 8630318 14 0.00% 0.00% 0.00% 0 hrpc -> request

39 11 5 2200 0.00% 0.00% 0.00% 0 IF-MGR control p

76 0 1 0 0.00% 0.00% 0.00% 0 HRPC hcomp reque

77 467922 208496012 2 0.00% 0.00% 0.00% 0 ppc405ex_emac_pr

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

43 25 6411 3 0.00% 0.00% 0.00% 0 Transport Port A

79 0 3 0 0.00% 0.00% 0.00% 0 HRPC hdm non blo

80 0 2 0 0.00% 0.00% 0.00% 0 HRPC hdm blockin

81 114181 8515129 13 0.00% 0.00% 0.00% 0 HIPC bkgrd proce

82 0 1 0 0.00% 0.00% 0.00% 0 RTTYS Process

83 12777 207688 61 0.00% 0.00% 0.00% 0 Hulc Port-Securi

84 0 1 0 0.00% 0.00% 0.00% 0 HRPC hpsecure re

85 0 1 0 0.00% 0.00% 0.00% 0 HRPC hlfm reques

46 11 109 100 0.00% 0.00% 0.00% 0 EEM ED Identity

87 13810548 42313057 326 0.00% 0.00% 0.00% 0 HLFM aging proce

88 39525361202755608 3 0.00% 0.00% 0.00% 0 HLFM address ret

89 0 1 0 0.00% 0.00% 0.00% 0 HRPC hrcmd reque

90 55276 386867 142 0.00% 0.00% 0.00% 0 HRPC hulc misc r

48 1150294 7875638 146 0.00% 0.00% 0.00% 0 EEM ED ND

92 112435 14198626 7 0.00% 0.00% 0.00% 0 HVLAN main bkgrd

93 0 2 0 0.00% 0.00% 0.00% 0 HVLAN Mapped Vla

94 0 2 0 0.00% 0.00% 0.00% 0 Vlan shutdown Pr

51 5 8 625 0.00% 0.00% 0.00% 0 USB Startup

96 0 7 0 0.00% 0.00% 0.00% 0 HULC VLAN REF Ba

97 0 1 0 0.00% 0.00% 0.00% 0 HRPC ilp request

98 9477986 42312941 223 0.00% 0.00% 0.00% 0 Hulc ILP Alchemy

99 0 1 0 0.00% 0.00% 0.00% 0 HULC PM Vector P

100 0 1 0 0.00% 0.00% 0.00% 0 HPM Msg Retry Pr

101 11262 709939 15 0.00% 0.00% 0.00% 0 DHCPD Database

53 0 1 0 0.00% 0.00% 0.00% 0 HULC ACL Tcam Me

103 4896 145542 33 0.00% 0.00% 0.00% 0 HPM Stack Sync P

104 0 1 0 0.00% 0.00% 0.00% 0 HRPC pm request

105 0 3 0 0.00% 0.00% 0.00% 0 HPM if_num mappi

61 36207251566306620 2 0.00% 0.00% 0.00% 0 Draught link sta

107 0 1 0 0.00% 0.00% 0.00% 0 HRPC pm-counters

108 0 1 0 0.00% 0.00% 0.00% 0 hpm vp events ca

109 0 1 0 0.00% 0.00% 0.00% 0 HRPC hcmp reques

110 96 20323 4 0.00% 0.00% 0.00% 0 HCEF ADJ Refresh

111 0 1 0 0.00% 0.00% 0.00% 0 HACL Queue Proce

112 0 1 0 0.00% 0.00% 0.00% 0 HRPC acl request

113 5 88 56 0.00% 0.00% 0.00% 0 HACL Acl Manager

114 0 1 0 0.00% 0.00% 0.00% 0 HRPC aim request

115 0 1 0 0.00% 0.00% 0.00% 0 HRPC backup inte

116 873 142271 6 0.00% 0.00% 0.00% 0 OBFL TEMP obfl0

117 0 1 0 0.00% 0.00% 0.00% 0 HRPC cdp request

118 0 1 0 0.00% 0.00% 0.00% 0 HULC CISP Proces

119 0 1 0 0.00% 0.00% 0.00% 0 HRPC dot1x reque

120 0 1 0 0.00% 0.00% 0.00% 0 Dot1X Msg Retry

121 0 3 0 0.00% 0.00% 0.00% 0 HULC DOT1X Proce

69 234758 8566631 27 0.00% 0.00% 0.00% 0 HULC Thermal Pro

123 0 1 0 0.00% 0.00% 0.00% 0 HRPC system mtu

124 0 1 0 0.00% 0.00% 0.00% 0 HRPC rep request

125 0 3 0 0.00% 0.00% 0.00% 0 REP Helper Proc

126 0 1 0 0.00% 0.00% 0.00% 0 HRPC sdm request

127 0 1 0 0.00% 0.00% 0.00% 0 SMI MSG Retry Pr

128 0 1 0 0.00% 0.00% 0.00% 0 HRPC Smart Insta

129 3034254 208495995 14 0.00% 0.00% 0.00% 0 Hulc Storm Contr

130 0 2 0 0.00% 0.00% 0.00% 0 HSTP Sync Proces

131 0 1 0 0.00% 0.00% 0.00% 0 HRPC stp_cli req

132 0 1 0 0.00% 0.00% 0.00% 0 HRPC stp_state_s

133 0 2 0 0.00% 0.00% 0.00% 0 S/W Bridge Proce

134 0 1 0 0.00% 0.00% 0.00% 0 HRPC hudld reque

135 0 1 0 0.00% 0.00% 0.00% 0 HRPC vqpc reques

136 0 1 0 0.00% 0.00% 0.00% 0 HRPC hled reques

73 5786350 42312728 136 0.00% 0.00% 0.00% 0 hrpc -> response

138 899566 31233243 28 0.00% 0.00% 0.00% 0 HL3U bkgrd proce

139 0 1 0 0.00% 0.00% 0.00% 0 HRPC hl3u reques

140 0 1 0 0.00% 0.00% 0.00% 0 HRPC obfl reques

142 0 1 0 0.00% 0.00% 0.00% 0 HRPC dtp request

143 0 1 0 0.00% 0.00% 0.00% 0 HRPC show_forwar

144 0 1 0 0.00% 0.00% 0.00% 0 HRPC snmp reques

145 0 1 0 0.00% 0.00% 0.00% 0 HULC SNMP Proces

75 621449 8678960 71 0.00% 0.00% 0.00% 0 hrpc <- response

147 7229405 17028423 424 0.00% 0.01% 0.00% 0 HRPC qos request

148 0 1 0 0.00% 0.00% 0.00% 0 HRPC span reques

149 0 1 0 0.00% 0.00% 0.00% 0 HRPC system post

150 0 1 0 0.00% 0.00% 0.00% 0 Hulc Reload Mana

78 0 6 0 0.00% 0.00% 0.00% 0 HULC Device Mana

152 671 237123 2 0.00% 0.00% 0.00% 0 HCMP sync proces

153 0 2 0 0.00% 0.00% 0.00% 0 image mgr

154 0 1 0 0.00% 0.00% 0.00% 0 HRFS OIR Proc

155 1088810 14198614 76 0.00% 0.00% 0.00% 0 Power RPS Proces

91 0 5 0 0.00% 0.00% 0.00% 0 HRPC emac reques

157 10 4 2500 0.00% 0.00% 0.00% 0 HL2MCM

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

158 521 2077 250 0.00% 0.00% 0.00% 0 AAA Server

159 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc

160 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr

161 0 1 0 0.00% 0.00% 0.00% 0 AAA System Acct

162 0 1 0 0.00% 0.00% 0.00% 0 AUTH POLICY Fram

163 4667 104757 44 0.00% 0.00% 0.00% 0 Auth Manager

164 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B

165 387 142272 2 0.00% 0.00% 0.00% 0 IP Admin SM Proc

166 14293061 13277836 1076 0.00% 0.01% 0.00% 0 CDP Protocol

167 72945 52523 1388 0.00% 0.00% 0.00% 0 SpanTree Helper

168 5 67 74 0.00% 0.00% 0.00% 0 ENABLE AAA

169 0 1 0 0.00% 0.00% 0.00% 0 CMD HANDLER

170 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

171 12615868 54865682 229 0.00% 0.01% 0.00% 0 DHCP Snooping

172 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces

173 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework

174 0 1 0 0.00% 0.00% 0.00% 0 EAP Test

175 0 3 0 0.00% 0.00% 0.00% 0 CEF switching ba

176 3725315 29677283 125 0.00% 0.01% 0.00% 0 IP Input

177 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl

178 287120 82801906 3 0.00% 0.00% 0.00% 0 IP ARP Track

179 0 3 0 0.00% 0.00% 0.00% 0 ADJ resolve proc

180 0 2 0 0.00% 0.00% 0.00% 0 ADJ NSF process

181 0 1 0 0.00% 0.00% 0.00% 0 IPv6 ping proces

182 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework

95 6 39 153 0.00% 0.00% 0.00% 0 HRPC vlan reques

184 0 2 0 0.00% 0.00% 0.00% 0 REP Topology cha

185 0 1 0 0.00% 0.00% 0.00% 0 SMI Director DB

186 0 1 0 0.00% 0.00% 0.00% 0 SMI CDP Update H

187 0 1 0 0.00% 0.00% 0.00% 0 SMI Backup Proce

188 0 2 0 0.00% 0.00% 0.00% 0 SMI IBC server p

189 0 1 0 0.00% 0.00% 0.00% 0 SMI IBC client p

190 0 2 0 0.00% 0.00% 0.00% 0 SMI IBC Download

102 23958337 425994344 56 0.00% 0.03% 0.04% 0 hpm main process

192 1338 711339 1 0.00% 0.00% 0.00% 0 Spanning Tree St

193 15412 104779 147 0.00% 0.00% 0.00% 0 802.1x switch

194 16437 709935 23 0.00% 0.00% 0.00% 0 802.1x MDA Aging

195 0 1 0 0.00% 0.00% 0.00% 0 802.1x Webauth F

196 0 2 0 0.00% 0.00% 0.00% 0 DTP Protocol

197 0 1 0 0.00% 0.00% 0.00% 0 HRPC IPv6 Host r

198 0 2 0 0.00% 0.00% 0.00% 0 IPv6 Platform Ho

199 0 1 0 0.00% 0.00% 0.00% 0 HRPC dai request

200 0 1 0 0.00% 0.00% 0.00% 0 HULC DAI Process

201 0 1 0 0.00% 0.00% 0.00% 0 HRPC power down

202 0 1 0 0.00% 0.00% 0.00% 0 HRPC ip device t

203 0 1 0 0.00% 0.00% 0.00% 0 HRPC ip source g

204 0 1 0 0.00% 0.00% 0.00% 0 HULC IP Source g

122 0 1 0 0.00% 0.00% 0.00% 0 HRPC lldp reques

206 300749 42682034 7 0.00% 0.00% 0.00% 0 UDLD

207 3524 1422736 2 0.00% 0.00% 0.00% 0 Port-Security

208 0 2 0 0.00% 0.00% 0.00% 0 Switch Backup In

209 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track Pr

210 0 1 0 0.00% 0.00% 0.00% 0 Link State Group

211 10144 709935 14 0.00% 0.00% 0.00% 0 MMN bkgrd proces

212 21282 4259587 4 0.00% 0.00% 0.00% 0 Ethchnl

213 13719 52480 261 0.00% 0.00% 0.00% 0 VMATM Callback

214 59307 851760 69 0.00% 0.00% 0.00% 0 CEF background p

215 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP

216 0 1 0 0.00% 0.00% 0.00% 0 CEF RF HULC Conv

217 0 2 0 0.00% 0.00% 0.00% 0 XDR mcast

218 25728 4259588 6 0.00% 0.00% 0.00% 0 IP ACL XDR LC Ba

219 0 1 0 0.00% 0.00% 0.00% 0 IPC LC Message H

220 0 1 0 0.00% 0.00% 0.00% 0 XDR RP Ping Back

221 1056 355669 2 0.00% 0.00% 0.00% 0 XDR RP backgroun

222 0 1 0 0.00% 0.00% 0.00% 0 XDR RP Test Back

223 0 1 0 0.00% 0.00% 0.00% 0 Critical Auth

224 2153 99951 21 0.00% 0.00% 0.00% 0 TCP Timer

225 591 1004 588 0.00% 0.00% 0.00% 0 TCP Protocols

226 791873 42305124 18 0.00% 0.00% 0.00% 0 Socket Timers

227 3975 145328 27 0.00% 0.00% 0.00% 0 HTTP CORE

228 1165 533513 2 0.00% 0.00% 0.00% 0 Cluster L2

229 36026 4259590 8 0.00% 0.00% 0.00% 0 Cluster RARP

230 31581 5334076 5 0.00% 0.00% 0.00% 0 Cluster Base

231 14155422 84610254 167 0.00% 0.03% 0.00% 0 DHCPD Receive

232 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

233 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

234 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

235 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

236 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

237 0 1 0 0.00% 0.00% 0.00% 0 IP RIB Update

238 5 1 5000 0.00% 0.00% 0.00% 0 LICENSE AGENT

239 0 2 0 0.00% 0.00% 0.00% 0 REP LSL Proc

240 0 2 0 0.00% 0.00% 0.00% 0 REP BPA/EPA Proc

241 0 1 0 0.00% 0.00% 0.00% 0 HRPC dhcp snoopi

242 10830484 54517197 198 0.00% 0.00% 0.00% 0 HULC DHCP Snoopi

243 341369 2047630 166 0.00% 0.00% 0.00% 0 IGMPSN L2MCM

244 1845731 16284423 113 0.00% 0.00% 0.00% 0 IGMPSN MRD

151 0 1 0 0.00% 0.00% 0.00% 0 HRPC hrcli-event

246 0 1 0 0.00% 0.00% 0.00% 0 IGMPQR

247 0 2 0 0.00% 0.00% 0.00% 0 L2TRACE SERVER

248 12175482 45633763 266 0.00% 0.00% 0.00% 0 Inline Power

249 5192709 94419728 54 0.00% 0.00% 0.00% 0 Marvell wk-a Pow

250 298456 2047628 145 0.00% 0.00% 0.00% 0 MLDSN L2MCM

251 0 1 0 0.00% 0.00% 0.00% 0 MRD

252 0 1 0 0.00% 0.00% 0.00% 0 MLD_SNOOP

253 0 1 0 0.00% 0.00% 0.00% 0 HRPC hl2mcm igmp

254 0 1 0 0.00% 0.00% 0.00% 0 HRPC hl2mcm mlds

255 41 994 41 0.00% 0.00% 0.00% 0 LOCAL AAA

256 0 1 0 0.00% 0.00% 0.00% 0 EPM MAIN PROCESS

257 378241 13546 27922 0.00% 0.00% 0.00% 0 OBFL UPTIME obfl

258 0 2 0 0.00% 0.00% 0.00% 0 TPLUS

259 37360 2499 14949 0.00% 0.00% 0.00% 0 OBFL MSG obfl0

260 5 12 416 0.00% 0.00% 0.00% 0 crypto engine pr

261 26 4 6500 0.00% 0.00% 0.00% 0 Crypto CA

262 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL

263 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL

264 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc

265 0 1 0 0.00% 0.00% 0.00% 0 HRPC x_setup req

266 0 2 0 0.00% 0.00% 0.00% 0 REP Switch Helpe

267 0 1 0 0.00% 0.00% 0.00% 0 Licensing MIB pr

268 15 73 205 0.00% 0.00% 0.00% 0 VTP Trap Process

269 214 1032 207 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

270 0 2 0 0.00% 0.00% 0.00% 0 DHCP Security He

271 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

272 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF

273 0 1 0 0.00% 0.00% 0.00% 0 HCD Process

274 0 1 0 0.00% 0.00% 0.00% 0 HRPC cable diagn

275 0 2 0 0.00% 0.00% 0.00% 0 DiagCard1/-1

276 1008019 182126662 5 0.00% 0.00% 0.00% 0 PM Callback

277 2136538 14240951 150 0.00% 0.00% 0.00% 0 VLAN Manager

278 42 1 42000 0.00% 0.00% 0.00% 0 OBFL ENV obfl0

279 13503 3413132 3 0.00% 0.00% 0.00% 0 dhcp snooping sw

280 43448 709931 61 0.00% 0.00% 0.00% 0 OBFL VOLT obfl0

281 0 14 0 0.00% 0.00% 0.00% 0 EEM Server

282 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc

283 0 1 0 0.00% 0.00% 0.00% 0 tHUB

284 52894 73422 720 0.00% 0.00% 0.00% 0 Syslog

285 0 2 0 0.00% 0.00% 0.00% 0 STP FAST TRANSIT

286 0 2 0 0.00% 0.00% 0.00% 0 CSRT RAPID TRANS

287 5263 142305 36 0.00% 0.00% 0.00% 0 OBFL POE obfl0

288 12378781 734816 16846 0.00% 0.00% 0.10% 0 crypto sw pk pro

289 5289 23707 223 0.00% 0.00% 0.00% 0 SSH Event handle

290 0 16 0 0.00% 0.00% 0.00% 0 SNMP Timers

291 11363542 40177387 282 0.00% 0.00% 0.00% 0 IP SNMP

292 3242930 20101805 161 0.00% 0.00% 0.00% 0 PDU DISPATCHER

293 46561288 20294276 2294 0.00% 0.00% 0.00% 0 SNMP ENGINE

294 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6

295 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

296 487 115 4234 0.00% 0.00% 0.00% 0 SNMP Traps

297 40617701193366205 3 0.00% 0.00% 0.00% 0 RADIUS

298 0 13 0 0.00% 0.00% 0.00% 0 hulc cfg mgr mas

299 14767 80 184587 0.00% 0.00% 0.00% 0 hulc running con

300 277058 2840377 97 0.00% 0.00% 0.00% 0 OBFL I/O Buffer

Vignesh Rajendran Praveen · ‎08-12-2013

Hi Thomas,

At the time you collected the "show proc cpu sorted" output where the "%FRNTEND_CTRLR-1-MGR_TXQ_FULL" error los still being seen ?

Thanks & Regards,

Vignesh R P

Vignesh Rajendran Praveen · ‎08-12-2013

Hi Thomas,

Usually when ever you will see the RedEarth Rx Mana process consuming high CPU we will see the following error message:

"%FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller TX queue reached watermark level"

This process is used for Punted Packet Handling. It is normally used for Microprocessor communication process i.e. it does periodic polling on the I2C interface connected to Microprocessor Controller Unit. Microprocessor Controller Unit (MCU) and IOS does sync up on various applications data e.g. PoE, Stack Power etc.

So in this situation when the device is reloaded problem gets fixed.

But if you see this message even without the above mentioned process not consuming high CPU or even after reload the error logs keep appearing, then it means the HW on the switch has gone faulty. And needs replacement.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P

Thomas Yarger · ‎08-14-2013

Thanks!

Herbert Patrman · ‎08-14-2019

Hello!

We're currently running 12.2(55r)SE on a WS-C2960S-48LPS-L and the switch started writing the following messages:

Aug 9 07:30:19 xxxxxxxx 56: Aug 9 05:30:32.069: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:35:20 xxxxxxxx 57: Aug 9 05:35:32.910: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:40:21 xxxxxxxx 58: Aug 9 05:40:33.761: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:45:21 xxxxxxxx 59: Aug 9 05:45:33.788: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:50:21 xxxxxxxx 60: Aug 9 05:50:34.545: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 07:55:22 xxxxxxxx 61: Aug 9 05:55:35.196: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level
Aug 9 08:00:22 xxxxxxxx 62: Aug 9 06:00:35.286: %FRNTEND_CTRLR-1-MGR_TXQ_FULL: The front end controller Tx queue reached watermark level

...

until Aug 13 11:57:52 - then it was rebooted for changing power cord.

about 4hrs before (in between nothing!) also this messages have been received:

Aug 9 03:11:48 xxxxxxxx 54: Aug 9 01:12:01.573: %HARDWARE-3-MCU_I2C_READ_ERROR: I2C read failed in MCU and attached devices (e.g. Temperature Sensor) can't be read.
Aug 9 03:11:48 xxxxxxxx 55: Aug 9 01:12:02.574: %HARDWARE-3-MCU_I2C_BAD_DEV: I2C bad device address: 4 (alchemy:0)

As a consequent the switch turned of all power feeds to POE devices (12 IP cams class 3 with 15,4W per device)!

Some additional informations: this actual switch already has been replaced by RMA about 4 weeks ago because the old one suddenly began to shutdown itself completly. The new one also shows the same symptoms after some days. Powercord has been replaced and is pluged together with an other switch which doesn't show this problem.

The POE devices now have been attached to power injectors to get sure that this is not the trigger.

So is this a hardware problem caused by one or more PE devices or the switch itself or is it a software problem indeed?

Best regards,

Herbert