No connection to site message on TBA

[Albert Wilhelm]

Hello,

I am getting a red highlight around the TBA for customer 1083. The logs indicate the TBA is not able to communicate back to the site - I am assuming the Thunderbolt portal.

Here is the event message:

Event ID                                         343540

Event Type                                     SERV.CPE.CONN

SERV.CPE.CONN                          2010-04-07 01:06:49.151499-05

Event Description                           CPE Connection, server detected

Event Message                              Site Comms down: 99.49.39.196

Customer Id                                  1083.ciscovar.com

Device ID                                      00:1A:70:8E:25:23

CONN, CLOSE or HEARTBEAT     CLOSED

UP or DOWN                                DOWN

Is the TBA telling me it can not communicate with the "mother ship"? :-)

I have had this error for the last few days, but the TBA was fine prior to that. Is there a way to find out if the TBA is communicating or know to the portal as offline?

Thank you.

Bert

[Marcos Hernandez]

Hi Bert,

Was there any change in your network that could be blocking traffic to the Portal? What is the status of the appliance itself? What is the status of the LED's? Can you find it and browse to its administration page?

Thanks,

Marcos

[Albert Wilhelm]

Marcus,

Thank you for your response.

1. There was no change to the network that would prevent the TBA from accessing the internet.

2. The portal indicates the TBA with a red highlight indicating down with the message on my first post.

3. The following LEDs are on and green: Power, Ethernet (flashing) and USB

4. Just tried to access the TBA's web GUI and got this error message:

Fatal error: Uncaught exception 'Exception' with message 'Unable to read configuration file' in /usr/local/htdocs/admin/classes/config.inc:39 Stack trace: #0 /usr/local/htdocs/admin/classes/config.inc(29): config->__construct() #1 /usr/local/htdocs/admin/pages/login.inc(31): config::singleton() #2 /usr/local/htdocs/admin/classes/page.inc(96): login->preprocess() #3 /usr/local/htdocs/admin/index.php(86): page->render() #4 {main} thrown in /usr/local/htdocs/admin/classes/config.inc on line 39

5. I unplugged the Ethernet cable to power cycle the TBA. The diags LED comes on and then the LEDs in #3 stay on. When I http to TBA, I still get the error message in #4.

6. Pinging the TBA works and responses.

Are you able to see the TBA? Do I need a new one?

Thank you.

Bert

[Marcos Hernandez]

I think we will have to RMA you a new one. Let me confirm and I will get back to you ASAP.

Thanks,

Marcos

[Marcos Hernandez]

Bert, can please you try a factory default  and reactivation?

[Albert Wilhelm]

Hello Marcus,

I performed the steps you requested to reset the TBA to factory default.

When the TBA resets after I released the Reset button, the Diags LED keeps flashing and I am not able to access the unit via the browser.

Do we need to RMA the unit?

Thank you for your support.

Bert

[Marcos Hernandez]

Hi Bert,

We will process an RMA. I apologize for the inconvenience.

Thanks,

Marcos

[Michael Holloway]

The flashing diag light may indicate that the USB key isn't properly seated.  We've seen this happen quite a bit with this case, you might just try giving it a hard push.  Should take about 10 minutes after that before the unit finishes pulling down a factory image and is becomes accessible via the web.

-mike

[Albert Wilhelm]

Hello Mike,

I did re-seat the flash card and reset the TBA. Unfortunantely, the TBA Diags LED is flashing and it has all night. So, I think this unit is toast. :-(

[Albert Wilhelm]

Just wanted to update this thread. The problem was in the TBA. Marcos sent me a new one and it works.

Problem solved. Thanks Marcos.

Bert

[Michael Holloway]

Bert,

Thanks for the update.

-mike

[netguyz08]

I actually had my two TBA's go offline, one had done it a few days ago, and I noticed because I wasn't getting reports lately. Another site went down at the beginning of May. Powercycling both of them brought them up and they got firmware 13.100 and are working again.

[Michael Holloway]

Hi Edward, this doesn't sounds like an issue we've seen before.  Can you provide the customer numbers so that we can take a look through our logs?  What were the symptoms you noticed on the portal before reseting the units?

I'm wondering if it's related to the issue at https://supportforums.cisco.com/message/3107704#3107704, with the customer numbers of the sites we should be able to tell.

-mike

[netguyz08]

Mike,

That other forum post is a possibility. I didn't see what firmware they had before, but power-cycling corrected it since they were able to go out and grab 13.100 and are now reporting that they are online again.

The ID's are 1080 and 1086 so you can take a look.

Both sides had similar messages before going down:

Severity	Warning
Type	SERV.CPE.CONN
Message	Site Comms down: xxx.xxx.xxx.xxx
Device ID	xx:xx:xx:xx:xx:xx
CONN, CLOSE or HEARTBEAT	HEARTBEAT
UP or DOWN	DOWN

[Michael Holloway]

Edward,

Looks like an unrelated issue to that other forum post.  I don't believe there is any forensic evidence for us to look at for these 2 sites, other than the loss of heartbeat event that the portal detected when they each went offline.  The fact that they upgraded themselves and restored their connection to the portal after a power cycle seems to indicate that the cause was with the TBA itself, but I can't say what might have locked them up in this manner, we haven't observed this failure mode before.  These are effectively linux boxes, and could be susceptible to locking up due to a power spike.  Are these devices being fed by clean UPS power?

-mike