VCS Expressway X8.5.3 became unresponsive and required hard reboot

[Paul Richardson]

Hi,

I have a client with a VCS Expressway appliance running X8.5.3 s/w that became unresponsive earlier today and required a hard reboot.  The event logs have many repetitions of the line "kernel: INFO: rcu_sched self detected stall on CPU"

I have checked on Cisco Bug Search for items that are still open and also issues rectified in X8.6, but nothing springs up.  This Expressway is not clustered and is configured in a DMZ with Advanced Networking Option and is setup as in the 3 port firewall model shown in Appendix 4 of the Deployment Guide.  As the unit is not under a service contract (despite our Service Delivery Manager reminding them) and the warranty period has expired, I am unable to open a TAC case for this problem.

Has anyone dealt with this issue, and if so how was it resolved?

Cheers, Paul Richardson

[Acevirgil de Ocampo]

Hi Paul,

Can you tell exactly what's going on your VCS appliance.

1. LED's status (alarms)

2. LCD display (is there a display)

Can you access the box using its IP address via web/SSH/telnet?

Regards,

Acevirgil

[Paul Richardson]

Hi Acevirgil,

The appliance was rebooted by the clients IT staff, so I don't know what the status of the LED's was at the time.  Apparently it was not accessible via web or SSH (telnet is turned off) at the time.

Cheers, Paul R.

[Acevirgil de Ocampo]

Hi Paul,

It's best way to do troubleshooting is seeing the box physically. If you are unable to access using its IP address, another way is to connect a console cable or keyboard and monitor using the USB and VGA port.

Using putty or other tool to console the box, use the following parameters:

If there is no display on the LDC panel of the VCS while powered on, there is a possibility the VCS image is corrupted or not booting on the right image.

Follow these procedure to boot the VCS in another image. This will reinstall the original firmware image (factory default). Have a copy of the release key of your VCS X8.x.x and you will be needing this if you want to upgrade again. Option keys and configurations will not be affected.

1. While successfully connected via console, reboot the box using the hard power switch located at the rear.

2. During boot up you will see an option where to boot the VCS. You will see image [TANDBERG 1/TANDBERG 2]. Select the other image using the arrow keys on your keyboard and hit enter. This should be executed quickly because the VCS will try and continue booting on the currently selected image.

3. The VCS will boot on the selected image and will restore the firmware to the original version.

4. Upgrade the VCS firmware if necessary using the correct release key.

This procedure will restore your VCS and make it work normally again.

I hope this helps.

Regards,

Acevirgil

[Paul Richardson]

Hi Acevirgil,

The unit has been up for 7 days now with no further errors in the event logs.  I am keeping it under observation.

Cheers

[vikram.pathania]

Hi Paul,

I have the same issue. Could you please share the Cisco Bug ID

Regards,

Vikram

[ahmed]

Hi Acevirgil,

it means vcs has 2 linux OS with 2 firmware or it just same single linux OS with 2 firmware,

so when i change other image means i change also entire VCS ?

Regards,

ahmad

[Wayne DeNardi]

Hi Ahmad,

It is essentially two different OSes installed, but only one can be active at a time. If one gets corrupted, you switch it back to the other. But it will likely be the previous software release unless, when you upgrade, you do it twice to make both the installed versions the same. 

Wayne