WS-C3750X-24S on 12.2(58)SE2 CPU problems hl3mm

[Bilal Nawaz]

Hey Guys, 3750x (2 in a stack - multiple SVI's, VLANs, ACLs) came shipped with 12.2(58) and had been put in production... Had several months without problems, however late last week there were a few trunks/portchannels added for new fit out floors, more multicast traffic too - the switch is ramping up on CPU to around 80 - 90%.

Yesterday CPU peaked 100% and dropped OSPF neighborships with a pair of ASR's at the WAN edge. Did cause a blip while the adjacencies got built again. Since the Cisco IP Phones are really flakey it was quite noticable that something had happed. But obviously the monitoring tools poll every x minutes and isnt real time.

I have already loaded c3750e-universalk9-mz.122-55.SE7.bin on the flash ready to be power cycled tonight and bring up this IOS.

Anyone see any problems with this?

Anyone experienced this before?

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

[InayathUlla Sharieff]

Bilal,

I think thats the known issue with this firmware. You can see many posts from Leo and me saying to use 122-55.SE7 to use on Cisco 3750 model.

Unless you need features in 12.2(58)SE or 15.0(1)SE or 15.0(2)SE, then I suggest you stay as far away from these.

For 3750E please use 122-55.SE7 as this is most stable and good software for this model.

HTH

Regards

Inayath

[Bilal Nawaz]

Thank you for confirming Inayath, I normally try and stick to 12.2 55 myself anyway.... Are there any bugs specifically related to hl3mm though - just curious why it would just shoot up like that for no reason. Apart from the fact of additional portchannel... And slightly more traffic flow.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

[InayathUlla Sharieff]

I can check that for you.

By any chance do you have output of show proc cpu during that time?

regards

Inayath

[Bilal Nawaz]

Unfortunately I couldn't get the show proc cpu during the peak, but I can show you what its like right now, sitting around 80% - for a switch stack thats not doing a great deal :-) .... see attached.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

[Bilal Nawaz]

Now that most users have gone home and the network has settled down, its sitting there quietly around 20 - 30%

ET-3750X-121#show proc cpu sorted

CPU utilization for five seconds: 26%/1%; one minute: 31%; five minutes: 22%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

125     4156175     3098829       1341  5.43%  5.08%  4.45%   0 hpm main process

372      194361      262380        740  2.39%  0.35%  0.20%   0 IP SNMP         

  10      146543        1267     115661  2.39%  0.29%  0.18%   0 Licensing Auto U

374     3431904      313484      10947  2.23%  0.35%  0.21%   0 SNMP ENGINE     

214     2383054     8354164        285  1.27%  0.68%  0.57%   0 IP Input        

  12      219350      248412        883  0.95%  0.62%  0.72%   0 ARP Input       

129      753299      121490       6200  0.79%  0.89%  0.87%   0 hpm counter proc

381      195992      676887        289  0.63%  0.10%  0.06%   0 IGMP Input      

373       34206      129074        265  0.63%  0.07%  0.01%   0 PDU DISPATCHER  

124       30247       10144       2981  0.63% 11.18%  3.12%   1 SSH Process     

  85      562244      278942       2015  0.63%  0.58%  0.57%   0 RedEarth Tx Mana

169      608417     1516804        401  0.47%  0.51%  0.38%   0 Hulc LED Process

  91      292617       27014      10832  0.47%  0.31%  0.32%   0 Adjust Regions  

232      190422      651990        292  0.31%  0.21%  0.10%   0 Spanning Tree   

213       13677     1751429          7  0.15%  0.01%  0.00%   0 IP ARP Retry Age

382      149593     1531975         97  0.15%  0.05%  0.05%   0 PIM Process     

  92        7499       73688        101  0.15%  0.01%  0.00%   0 hrpc -> response

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

[InayathUlla Sharieff]

Bilal,
There are couple of things which to be check during the issue. Like the SDM usage ,routing usage, multicast...etc...

Could of bugs to name:

CSCsm98500    3750;High CPU hl3mm proces even after multicast-routing vrf removed 
CSCsq20951    TEA 3750;High CPU hl3mm proces even after multicast-routing vrf remove 

CSCtn42790    3560X/3750X: Elevated CPU usage due to Hulc LED process
CSCtu15860    high cpu in dot1x with MDA and data vlan equals to voice vlan
HTH
Regards Inayath

[Leo Laohoo]

Hi Bilal,

Even though I don't know your network or your stack configuration, hands down, 12.2(55)SE7 is far superior to 12.2(58)SE (and later). 

[Bilal Nawaz]

Thanks guys. I checked the SDM usage and we were nowhere near hitting any limits at all (during high CPU). I managed to downgrade last night but with some trouble.
Used the .bin IOS on flash and set the command, boot system switch all flash:xxxxxxx.bin.
I was SSH'd on to the stack from home, did the reload and only the master came up, the second switch didn't come up at all... Went in to check it out at midnight and second switch was powered on, but not responding at all. Fan lights were on at the back of the switch, but not even the console light was lit at the back. Consoled on to it too and still nothing.
Until I decided to give it a kick (not literally), took the stack power out and the power lead, plugged it back in and then it managed to boot properly.
Just to double make sure, I rebooted the stack again afterwards, and both came back fine.

Strange.... We'll see how it behaves today :-)

Thanks once again

Sent from Cisco Technical Support iPhone App

[Leo Laohoo]

Keep us posted. 

[Bilal Nawaz]

This is probably a great example of what it means to downgrade to 12.2(55) - and absolutely great advice .... The difference between IOS images from 12.2(58) most left of the charts. The gap is when I downgraded to 12.2(55) and then had to go in to reboot and fix up the stack again. Quite self explanitory!

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

[Leo Laohoo]

Nice one, Bilal!