CISCO EX60 - after switch off the system rebooting loop

christianfroehlich · ‎09-12-2012

Dear Community,

The problem only occur with TC5.1.4 / Touchpanel installed and after switching the system off it reboot may several times. These Factory Resets has been done:

Factory reset using the Touch Panel

1. Select More from the main panel.

2. Select Settings > Administrator settings and select Reset.

3. Select Factory Reset.

and

Factory Reset mechanically EX 60

1. Unplug the power cable.

2. Replug the power cable.

3. Press and hold the power button immediately when the green LED in the bottom left corner lights up. Hold until the LED turns off and on again after approximately 10 seconds.

4. Release the power button, and press it twice within 2 seconds.

Is that a hardware Issue with the EX60 or the Touchpanel? Please see enclosed log

Shilpa Shree Settikere Nanjundappa · ‎09-12-2012

Try logging into the EX60 using root login and issue the command 'touch /tmp/noboot'.

See if this stops the constant reboot.

Thanks,

Shilpa.

Danny De Ridder · ‎09-12-2012

Hello Christian,

thanks for adding the logfiles of your crash(es). Looking at those file I can see there's an audio crash. Check the following files :

console

Sep 12 10:56:41 (none) audio0: uhpidsp: tx-buffer not free for 10 tries

Sep 12 10:56:42 (none) audio0: uhpidsp: FATAL: communication with SYSTEM_MAIN is dead

Sep 12 10:56:43 (none) audio0: uhpidsp: *** UHPI driver state ***

Sep 12 10:56:44 (none) audio0: uhpidsp: Interface is alive

Sep 12 10:56:45 (none) audio0: uhpidsp: Rx buffers pending: no

Sep 12 10:56:45 (none) audio0: uhpidsp: Rx Counter: 159, Tx Counter: 3117

Sep 12 10:56:46 (none) audio0: uhpidsp: Dsp to host buffer state:

Sep 12 10:56:46 (none) audio0: uhpidsp: packet, status, length, dT (uS)

Sep 12 10:56:46 (none) audio0: uhpidsp: [ 0] 3104, used, 36, 10014

Sep 12 10:56:46 (none) audio0: uhpidsp: [ 1] 3105, used, 36, 9987

Sep 12 10:56:46 (none) audio0: uhpidsp: [ 2] 3106, used, 36, 10015

Sep 12 10:56:47 (none) main: HEARTBEAT: SYSTEM_AUDIO0 not responding at 123.896454425 secs

Sep 12 10:56:47 (none) main: HEARTBEAT:

audio0.log

Sep 12 10:56:41 (none) audio0: uhpidsp: tx-buffer not free for 10 tries

Sep 12 10:56:42 (none) audio0: uhpidsp: FATAL: communication with SYSTEM_MAIN is dead

Sep 12 10:56:43 (none) audio0: uhpidsp: *** UHPI driver state ***

Sep 12 10:56:44 (none) audio0: uhpidsp: Interface is alive

Sep 12 10:56:45 (none) audio0: uhpidsp: Rx buffers pending: no

Sep 12 10:56:45 (none) audio0: uhpidsp: Rx Counter: 159, Tx Counter: 3117

Sep 12 10:56:46 (none) audio0: uhpidsp: Dsp to host buffer state:

Audio board within the codec lost commmunications with the main() application running on the codec's main PowerPC[PPC]. If audio board is "sane" it sends heartbeats. If communications between the audio board and PPC, those heartbeats stop and we get to see the above messages printed in bold in the logs.

We have tools which match crashes using "signatures". When I ran the tool, it came back with the signature : "Audio0 complains main is dead, main complains audio0 is dead", but there's no matching ddts.

Do not know whether this is hardware or software at this point. Will check with engineering and get back to you.

One can use the method 'touch /tmp/noboot' to avoid the codec reloading, but that's more to be able to capture data. You already sent the logs and did all the right things!

Regards,

Danny.

Danny De Ridder · ‎09-13-2012

Hello,

talked to escalation team in Oslo and they came back with a few possible software bugs which could explain the crash you are seeing. The references for these issues are the following :

Bug 127686 - Cyclic reboot - SYSTEM_AUDIO0, lost heartbeat

Bug 129528 - Charlie: Floating serial line hack is broken on Pluto

These issues are fixed in TE6 alpha images.

Would you be willing to try such an image to verify your issue is resolved?

If so, I will make image and release key available to you.

Danny.

Danny De Ridder · ‎09-13-2012

Hello,

I did not read your original post not good enough.

You explained that only when you switch off the EX60, it goes into a cyclic reboot ?!

So it is not a crash when system is idle or so, but when we try to shut it down. I assume that is via the power button?

Strange things I see is following on the touch log :

Jan 1 00:01:43 endeavour main: TT3.1.4.295090

Jan 1 00:01:43 endeavour main: SW Release Date: 2012-08-10

Jan 1 00:01:43 endeavour main: Product information

Jan 1 00:01:43 endeavour main: Endeavour

Jan 1 00:01:43 endeavour main: Main board: E

Jan 1 00:01:43 endeavour main: Using server ip:169.254.1.1, and port: 65008

Jan 1 00:01:43 endeavour main: Connected to tshell

Sep 12 10:56:26 endeavour main: no cradle

Sep 12 10:56:59 endeavour main: Exit from signal handler with 0

Does this mean shutdown was started via touch panel? Do not know how one can do that. See only restart as an option.

Another thing I see in the main log is :

Sep 12 10:56:12 (none) main: Mcfsm::handleAnyMessage() unhandled msg EPAppl_Command_Req from PARAGENT

Sep 12 10:56:12 (none) main: VLCSubscriber::SetOutputPresentStatus(): unknown output port instance

Sep 12 10:56:20 (none) main: VLCSubscriber::SetOutputPresentStatus(): unknown output port instance

Sep 12 10:56:47 (none) main: HEARTBEAT: SYSTEM_AUDIO0 not responding at 123.896454425 secs

Sep 12 10:56:47 (none) main: HEARTBEAT:

Normally I would expect to see a power button event or so and main with exit code 0, but main crashes :-(

/bin/tandberg: line 24: 2357 Aborted (core dumped) $cmd /apps/main "$*"

Main has stopped with exit code 134

Shutting down services...

On my EX90 when I ask to reboot I get :

Sep 13 10:42:28 ppc main: CUIL reboot requested

Sep 13 10:42:28 ppc main: bootnotifier: Boot is requested

Can you please explain how you shut down the system???

christianfroehlich · ‎09-13-2012

They switch off / shut down on EX 60 direct

Danny De Ridder · ‎09-13-2012

I think there's an issue with the power button on your EX60 which triggers some unknown event sent to the main application which crashes. Can you, as an alternative, ssh into your machine and do following shutdown procedure :

[dderidde-ex90-home:~] $ tsh

Welcome to dderidde.ex90.home

TANDBERG Codec Release TC5.1.4.295090

SW Release Date: 2012-08-10

OK

xcommand Boot Action:shutdown

That should shutdown your system in an orderly fashion WITHOUT using the power button as trigger.

christianfroehlich · ‎09-13-2012

On the Touchpanel the message appear like "Software update" "Downloading Software"

christianfroehlich · ‎09-13-2012

I send out exchange EX60

Danny De Ridder · ‎09-13-2012

I would however check whether the system shuts down properly using the API instead of the power button, just to make sure.