All APP1000 devices down/red at 11:55ET 5/18

[davebainum]

Hi all,

Was wondering if it's just me - all three of our APP1000's seem to be down/red simulaneously as of about 3-4 minutes ago - and keeping going up and down every minute or so...

All three are at different locations and on three different circuits (although ironicallly, all three are with Cox Business Internet).  The sites do still have Internet access though.

Regards,

-- Dave Bainum (dbainum@ritetech.net),  PMP*
[PMP=PMI Certified Project Management Professional]
<Interested in great RP & other IT deals?? See tinyurl.com/plf8wz>
RiteTech LLC / www.ritetech.net / +1 (703) 561-0607
Creators of the www.RPConnect.net suite of applications

[Marcos Hernandez]

Dave,

It seems that all accounts went through a maintenance window at the same time, which is incorrect. You were not the only one affected. I still need engineering to take a look at this to confirm. We will get back to you tomorrow morning.

Thanks,

Marcos

[Marcos Hernandez]

Actually, I stand corrected. The Event History is not showing the TBA having gone through a maint. window. The tunnels simply went down. We will investigate and update everybody.

Marcos

[Michael Holloway]

Hi folks,

We believe we've identified and corrected the issue that caused the bouncing up and down of sites last night.  The problem appears to have been with a load balancer at the portal that distributes heartbeat load across multiple processes.  When a single customer site was activated yesterday, it pushed the number of active heartbeat sessions over a threshold that triggered the bug and caused the heartbeat instance to die.  This in turn pushed load onto other instances, causing them to eventually experience the same bug and also die.

A fix has been applied and we don't expect to see this issue again, but we'll keep a close eye on it.

-mike

[davebainum]

Ok, thanks for the updates.

-- Dave  Bainum (dbainum@ritetech.net),  PMP*
[PMP=PMI Certified Project Management Professional]
tinyurl.com/plf8wz>
RiteTech LLC / www.ritetech.net / +1 (703) 561-0607
Creators of the www.RPConnect.net suite of  applications