c3950: Intermittent msg "%SYS-2-INTSCHED: 'may_suspend' at level 4"

[portmannth]

Hello,

For a week now, one of my Cisco 3945 routers displays, from time to time, the following message in its log:

2012-07-16T14:12:43.852017+00:00 cb00-r103 220: [syslog@9 s_sn="59"]: Jul 16 16:12:42 MEST: %SYS-2-INTSCHED: 'may_suspend' at level 4  -Process= "OSPF-100 Hello", ipl= 4, pid= 320

2012-07-16T14:12:43.852017+00:00 cb00-r103 221: [syslog@9 s_sn="60"]: -Traceback= 15CB073z 9081B7z 2851571z 11B7582z 11C275Ez 11C2906z 11C03E3z 1B907B5z 1B9064Cz 1BA5737z 172047Fz 47FFDCz 4D2EEDFz 4D2ED71z 4D2EA2Bz 4D30097z

2012-07-16T15:06:11.686817+00:00 cb00-r103 222: [syslog@9 s_sn="61"]: Jul 16 17:06:10 MEST: %SYS-2-INTSCHED: 'may_suspend' at level 4  -Process= "OSPF-100 Hello", ipl= 4, pid= 320

2012-07-16T15:06:11.686817+00:00 cb00-r103 223: [syslog@9 s_sn="62"]: -Traceback= 15CB073z 9081B7z 2851571z 11B7582z 11C275Ez 11C2906z 11C03E3z 1B907B5z 1B9064Cz 1BA5737z 172047Fz 47FFDCz 4D2EEDFz 4D2ED71z 4D2EA2Bz 4D30097z

2012-07-16T15:35:49.632388+00:00 cb00-r103 224: [syslog@9 s_sn="63"]: Jul 16 17:35:48 MEST: %SYS-2-INTSCHED: 'may_suspend' at level 4  -Process= "OSPF-100 Hello", ipl= 4, pid= 320

2012-07-16T15:35:49.632388+00:00 cb00-r103 225: [syslog@9 s_sn="64"]: -Traceback= 15CB073z 9081B7z 2851571z 11B7582z 11C275Ez 11C2906z 11C03E3z 1B907B5z 1B9064Cz 1BA5737z 172047Fz 47FFDCz 4D2EEDFz 4D2ED71z 4D2EA2Bz 4D30097z

2012-07-17T04:05:53.765213+00:00 cb00-r103 226: [syslog@9 s_sn="65"]: Jul 17 06:05:52 MEST: %SYS-2-INTSCHED: 'may_suspend' at level 4  -Process= "OSPF-100 Hello", ipl= 4, pid= 320

2012-07-17T04:05:53.765213+00:00 cb00-r103 227: [syslog@9 s_sn="66"]: -Traceback= 15CB073z 9081B7z 2851571z 11B7582z 11C275Ez 11C2906z 11C03E3z 1B907B5z 1B9064Cz 1BA5737z 172047Fz 47FFDCz 4D2EEDFz 4D2ED71z 4D2EA2Bz 4D30097z

It never happened before and the configuration did not change. The only thing that happened, the WAN connection (point-to-point to a Cisco 3845) went down and the router rebooted while the WAN was down. When the WAN came up again, everything went fine, until about an hour later and the first occurence of this mentioned log.

This 3945 does establish an IPSec tunnel with its peer (the 3845) and all the trafic, including OSPF, is going through the tunnel.

Any idea on why this message appears and what it really means, because the output interpreter tools is not really usefull.

Thanks in advance

Tom

[Giuseppe Larosa]

Hello Tom,

the message should be related to the interrupt scheduler, however I would focus on the presence of a traceback with a dump of exadecimal stream

This is usually the sign of a SW bug, because it means a routine or other piece of code is not working properly.

An IOS upgrade may be recommended

Hope to help

Giuseppe

[portmannth]

Hello Giuseppe,

So, if it is a software bug, why did it not show up before? Or did the image get corrupted on the flash, so the flash might be bad? Maybe it was only a bad read and it can be resolved with a reboot (I might try that tomorrow, thursday)? Or it might be the RAM. Is there a way to test these different parts without affecting the users?

But I still do not understand what this "may suspend" means, what if I let it run like that, doing nothing?

Thanks

Thomas

[Giuseppe Larosa]

Hello Thomas,

the scheduler is that part of OS that dedicates cpu timeslots to processes, what is happening is that a specific process ,OSPF in your case, is taking "too much" time, the scheduler could suspend the involved process in an attempt to avoid to have cpu resources taken by the process for more then a defined treshold / time interval

You could try to increase the time interval with the command

see

http://www.cisco.com/en/US/docs/ios/12_4t/12_4t2/intrnask.html

try to increase

scheduler interrupt mask time

I don't think the IOS image is corrupted, and as I wrote a traceback is  a sign of a SW bug. As any SW bug there are specific conditions that act as a trigger for the SW bug to be seen.

The risk is that OSPF adjacencies may flap if the process is suspended by the scheduler and this likely would lead to new chances for OSPF process to be suspended again.

Edit:

the command I was thinking of is actually scheduler interval I would try this one

see

http://www.cisco.com/en/US/docs/ios/12_0/configfun/command/reference/frgenral.html#wp1019389

Hope to help

Giuseppe

[portmannth]

Hello Giuseppe,

I restarted the router yesterday evening and since had two tracebacks again. These next two weeks, I will let a colleague follow the status of this case in collaboration with our Cisco partner.

tom

[pedram]

Hello

What about the following message ?

%SYS-2-INTSCHED: 'may_suspend' at level 4  -Process= "Per-minute Jobs", ipl= 4, pid= 87
-Traceback= 30058264z 35E25D34z 35E26018z 35E23540z 357CEE38z 35E27FF0z 35E28094z 371F3F4Cz 35E27E3Cz 300668C0z 3076C364z 300329D0z 300329B4z
000263:

[subash cobi]

Hi Pedram,

I'm getting the same message as you.

Can't seem to find much information on it.

Did you manage to identify what is the cause of this message.