higher than normal cpu for fed (PunjectRx) process

CrackedJack1 · ‎02-17-2014

I noticed on my 3850 that the CPU seems to have gone up overnight a fed days ago and has stayed there. Everything seems to be fine at the moment but I was just wondering what the process is that's causing it (google gives me nothing) and if I should be worried.

Running show processes cpu sorted gives me this:

Core 0: CPU utilization for five seconds: 15%; one minute: 18%; five minutes: 25%
Core 1: CPU utilization for five seconds: 97%; one minute: 45%; five minutes: 34%
Core 2: CPU utilization for five seconds: 15%; one minute: 51%; five minutes: 39%
Core 3: CPU utilization for five seconds: 10%; one minute: 20%; five minutes: 42%
PID    Runtime(ms) Invoked uSecs 5Sec     1Min     5Min     TTY   Process
5711   183846      14637156 217    27.84    27.34    27.31    1088 fed
10184 481691      20810070 573    5.20     5.37     6.54     34816 iosd

I asked for detailed info on the fed process and got the output below. As you'll notice, the high cpu value on the core doesn't always show up on the same core. What is that PunjectRx process that's responsible for the higher than normal value do?

Core 0: CPU utilization for five seconds: 20%; one minute: 27%; five minutes: 30%

Core 1: CPU utilization for five seconds: 18%; one minute: 24%; five minutes: 28%

Core 2: CPU utilization for five seconds: 8%; one minute: 25%; five minutes: 34%

Core 3: CPU utilization for five seconds: 92%; one minute: 70%; five minutes: 52%

PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

(%) (%) (%)

5711 L 18656 146354252 217 26.9 27.2 27.3 1088 fed

5711 L 3 5711 29060 7853 0 0.00 0.00 0.00 1088 fed

5711 L 0 6144 2809206 23626670 0 0.00 0.01 0.02 1088 fed

5711 L 1 6145 5570 313407 0 0.00 0.00 0.00 1088 fed

5711 L 0 6146 296459 269367770 0 0.00 0.05 0.05 1088 CMI default xdm

5711 L 0 6147 3270 209596 0 0.00 0.00 0.00 1088 fed

5711 L 1 6150 2014171 254472866 0 0.34 0.30 0.29 0 fed-ots-main

5711 L 2 6151 600 13111 0 0.00 0.00 0.00 0 fed-ots-mo

5711 L 2 6153 3715892 3065842 0 0.48 0.48 0.48 0 fed-ots-nfl

5711 L 0 6154 326140 18702233 0 0.00 0.00 0.00 0 fed

5711 L 0 6155 137920 3126591 0 0.00 0.00 0.00 0 pdslib thread

5711 L 0 6156 1820820 117290277 0 0.00 0.02 0.02 0 Timer

5711 L 2 6157 0 2805 0 0.00 0.00 0.00 0 fed

5711 L 0 6176 290660 152790809 0 0.72 0.77 0.73 0 IntrDrv

5711 L 0 6179 486230 31194756 0 0.00 0.02 0.01 0 LED

5711 L 0 10160 60 9427 0 0.00 0.00 0.00 0 Xcvr

5711 L 0 10162 233781 204227456 0 0.24 0.24 0.24 0 Xcvr

5711 L 2 10686 50780 1309757 0 0.00 0.00 0.00 0 PdsEvent

5711 L 3 10687 2302060 113878739 0 24.0 24.0 24.0 0 PunjectRx

5711 L 2 10688 2689529 198752351 0 1.15 1.34 1.39 0 PunjectTx

5711 L 3 10689 0 7522 0 0.00 0.00 0.00 0 Consistency

5711 L 2 10690 0 7522 0 0.00 0.00 0.00 0 ShFwd

5711 L 0 10692 0 7544 0 0.00 0.00 0.00 0 eicore_bdn

5711 L 0 10693 0 7544 0 0.00 0.00 0.00 0 eicore_bdn

5711 L 0 10694 0 7544 0 0.00 0.00 0.00 0 eicore_bdn

5711 L 0 10696 50 7599 0 0.00 0.00 0.00 0 eicore_ipc

5711 L 3 10717 0 7752 0 0.00 0.00 0.00 0 fed

5711 L 0 10952 0 7849 0 0.00 0.00 0.00 0 fed

5711 L 0 11072 4281230 128013576 0 0.00 0.01 0.02 0 XcvrPoll

Gavin Barber · ‎03-06-2014

Seeing similar here on 3850 stack. Running 03.03.01SE IOS-XE.

Core 0: CPU utilization for five seconds: 45%; one minute: 40%; five minutes: 39%

Core 1: CPU utilization for five seconds: 57%; one minute: 46%; five minutes: 45%

Core 2: CPU utilization for five seconds: 58%; one minute: 57%; five minutes: 54%

Core 3: CPU utilization for five seconds: 39%; one minute: 56%; five minutes: 56%

PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

(%) (%) (%)

5679 L 2207615 162418091 66 23.7 23.1 22.7 1088 fed

5679 L 2 5679 29940 7879 0 0.00 0.00 0.00 1088 fed

5679 L 0 6112 321780 319764 0 0.00 0.03 0.04 1088 fed

5679 L 2 6113 520 24080 0 0.00 0.00 0.00 1088 fed

5679 L 2 6114 8770 5395188 0 0.00 0.03 0.04 0 CMI default xdm

5679 L 1 6115 320 16534 0 0.00 0.00 0.00 0 fed

5679 L 2 6118 1130370 5020432 0 0.62 0.74 0.74 0 fed-ots-main

5679 L 1 6119 20 1303 0 0.00 0.00 0.00 0 fed-ots-mo

5679 L 2 6121 3855140 224587 0 0.52 0.49 0.48 0 fed-ots-nfl

5679 L 1 6122 23500 1399239 0 0.00 0.04 0.01 0 fed

5679 L 2 6123 11430 231772 0 0.00 0.01 0.03 0 pdslib thread

5679 L 1 6124 189270 9882134 0 0.05 0.04 0.04 0 Timer

5679 L 1 6125 0 2808 0 0.00 0.00 0.00 0 fed

5679 L 1 6144 793970 8150598 0 0.29 0.21 0.20 0 IntrDrv

5679 L 1 6147 33730 2295636 0 0.00 0.01 0.00 0 LED

5679 L 0 10452 40 9474 0 0.00 0.00 0.00 0 Xcvr

5679 L 1 10456 1922330 107887089 0 0.33 0.22 0.20 0 Xcvr

5679 L 3 12275 110 10697 0 0.00 0.00 0.00 0 PdsEvent

5679 L 2 12276 980070 7963687 0 0.71 0.49 0.48 0 PunjectRx

5679 L 0 12277 1263190 4878455 0 21.1 20.8 20.5 0 PunjectTx

5679 L 0 12278 0 7567 0 0.00 0.00 0.00 0 Consistency

5679 L 0 12279 0 7567 0 0.00 0.00 0.00 0 ShFwd

5679 L 1 12283 0 7580 0 0.00 0.00 0.00 0 eicore_bdn

5679 L 1 12284 0 7580 0 0.00 0.00 0.00 0 eicore_bdn

5679 L 1 12285 0 7579 0 0.00 0.00 0.00 0 eicore_bdn

5679 L 3 12289 10 7608 0 0.00 0.00 0.00 0 eicore_ipc

5679 L 0 12298 0 7788 0 0.00 0.00 0.00 0 fed

5679 L 2 12610 0 7875 0 0.00 0.00 0.00 0 fed

5679 L 2 12801 233040 8635591 0 0.05 0.02 0.03 0 XcvrPoll

CrackedJack1 · ‎03-06-2014

I'm running 3.3.1 as well. I see your high usage is on transmit, not receive. Do you know if it was a sudden jump or more gradual or always like that?

Mine was sudden and in the middle of the night when nobody was working and I've not noticed any other issues which makes me think something in the switch has gone wrong.

I've opened a tac case but i'm not holding my breath. My environment has a lot of arp and broadcast traffic which, based on past experience, tends to be a red herring when it comes to troubleshooting and lots of time spent looking at false leads. I'm thinking reboot the stack but I would prefer to avoid that unless absolutely required.

If there's any info you can think of which might help me figrue this out i would appreciate it.

CrackedJack1 · ‎03-06-2014

Don't know if this is related but I've noticed my time is off by almost a minute on the switch. I've seen a note in my log saying time is running backwards on the 2nd switch in the stack and even though it says my NTP source is syncrhonized, it's about 50 seconds aheads of the real time.

An older cisco switch trunked off the 3850 which is getting NTP time from the same source as the 3850 is ok though. Just the 3850 is wrong but I don't know when that problem started (or how to fix it)

CrackedJack1 · ‎10-25-2014

Bringing back my old thread here.... I'm still having the problem but now the process which is using 20%+ under fed is PdsEvent.

At one point both PunjectRx and PdsEvent were high, now PunjectRx is down to normal and PdsEvent is up there.

Anyone know what PdsEvent is?

I'm running 3.3.3 at the moment.

Tzy Chun Chong · ‎01-11-2015

I got the same issue, i found this article is useful to find out the culprit switchport, but am not trying yet. use sharing purpose for you to try out. Let me know your outcome if possible.Thanks.

CrackedJack1 · ‎01-12-2015

Was there supposed to be a link in your post? Doesn't look like it came through.

At the moment my CPU is ok (around 35%) and both the PdsEvent and PunjectRx are good. I've looked at the CPU troubleshooting article but it didn't help me much.

Tzy Chun Chong · ‎01-12-2015

Opps.. My bad, yes I found this link.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html

My device that happened already power cycled (due to traffic intermittent) and problem resolved, but I managed to show-tech during the problem. From show-tech I found the iosd and fed CPU utilization are higher than normal. In my current status each iosd and fed only average in 3% CPU consumption but from the show-tech it surged to iosd 25% and fed 17% and the total CPU goes up to >60% and caused traffic intermittent. If the same problem happen again i gonna try this step to find out the culprit switchport.

Cheers

John Comfort · ‎09-12-2016

I experienced a similar problem where FED CPU was at 50+%, PDSD was around 30%. Switch uptime was 1yr, 14wks. Power cycling each switch individually (to eliminate downtime) corrected the problem. I started with the master switch 1, then switch 2 to reduce number of switchovers. Be sure to WAIT for all processes to sync. That is, stack members (show switch) will show Active/Member prior to HA sync. We want the stack members to show Active/Standby. After the HA sync process completes, the next switch is ready to be power cycled.

CPU is holding at <10% which is normal.

Kushal Singh Bisht · ‎03-24-2017

Want to know the reason for High utilization due to Fed(PunjectRx) .

what is PunjectRx actually ?