WiSM crash with no crash log

[Amjad Abdullah]

Guys,

I have many WiSM WLC's running 7.0.116.0. One WLC was rebooted few days ago but there was no crash file and nothing in logs say why this issue happened.

There was a power problem at the same time the WLC rebooted (some switches and PE's was rebooted as well) but if it is a power issue why only one WLC inside the WiSM rebooted and the other WLC is still working fine with no reboot?
I have 5 WiSM modules connected to the same 6500 box, only one WLC was rebooted which indicates a crash but no crash file registered for it.

Is there anyway I can find the reason why that WLC was rebooted?

Could it be a power issue over all? if yes what about my concerns above?

Thanks.

Amjad

[Justin Kurynny]

Amjad,

Power could definitely cause an issue like this. Do your 6500 logs have anything about loss of power to a power supply and/or powering down a module due to a low power condition?

Justin

[Amjad Abdullah]

Thanks Justin for your reply.

In my case it am really suspecious if it is a power issue. Because why it is only one WLC in the module that was rebooted? Why the issue did not reboot the whole module (2 WLCs)?

unfortunately I logged into the 6500 box and it does not have any logs at the period when the reboot happened.

[Justin Kurynny · Accepted solution]

Amjad,

What is the power supply model you have in your 6500?

Do you have 1 power supply or 2?

If 2, are you running them in combined mode or redundant mode?

These questions are specific to the behavior of the 6500 in the event of a partial power loss, such as a failing power supply or loss of current from the mains.

The following is an excerpt from the 6500 Operational Best Practices guide:

2.10 Power Shutdown Sequence

The previous section highlighted the recommendation for running in redundant mode. Should there be a requirement to run in combined mode, an extra precaution needs to be taken. In the event of a failing power supply that leaves the system without enough power to
drive the total configuration, inline devices and line cards will be shut down by the power management software to fall within operating power capabilities.

In order for the user to recognize the event and its consequences, the shutdown sequence logic should be known and understood: in particular, the Catalyst 6500 control software will first start to power down inline devices from the highest numbered port to the lowest numbered port, then from the bottom slot up. If after powering down all inline devices there is still insufficient power, the system will start to power down line cards from the bottom slot up with the exception of the supervisor module and of the services modules (that is, firewall module, VPN module, IDS module, and so on). Supervisor and services modules are the last modules left operating. For this reason, important hosts should be plugged into ports on the top-most modules (that is, slots 1, 2, 3, and so on) and from the lower numbered ports up. It is worth noting that the order of shutdown is fixed by the system and cannot be changed.

While this exceprt specifically says that SUPs and service modules are the last operating modules in a low power condition, I am curious if you had a condition wherein the 6500 made the decision to power down one of your wisms because power was too low to support all 5 plus the Sup(s).

Justin

[Amjad Abdullah]

Justin:

Thanks a lot. your answer is very useful to me and your quote from the document as well.

We are not confirming that restart was not due power issue.

But this does not answer the quesiton: "Can we know why the WLC was rebooted"?
I would say therei s no way to get this piece of info if there is no crash file or something useful in WLC logs.

I just want to make sure if my thoughts are correct or if there is somehow a way to figure out why the WLC restarted?

Thanks.

Amjad

[Justin Kurynny · Accepted solution]

Amjad,

The place to find logs about power being cut to your wism is going to be in the 6500 logs.

6500# show logging

Unfortunately, the default buffer size on catalyst switches tends to be pretty small by default and old events are bubbled out to make room for new ones.

I don't know of any way to find this information on the wism. A loss of power situation is one you rarely see in logs because after the device has lost power, it doesn't have the capacity to write a log event. On some platforms, such as desktop OSes, you will see at power-on a message saying something like "Windows did not shut down properly..." blah blah, but that log is generic and could arise from many conditions.

The best way to get this information in the future is to set up syslogging today. Send all your logs from the 6500 and from each of your wisms off-box to a third-party syslog collector.

Justin

[Amjad Abdullah]

Justin,

Thanks for your infor.

I looked for the logs in 6k buffer and found old logs only (2w3d old) but could not find newer (same or prevous dsay ) logs!! which is strange.

We are already setting syslog for all our devices. However, for this specific box it seems they forgot to configure it!! we are going to configure it very soon of course.

Your information are so useful. Now my concerns were answered. Many thanks.

Amjad