[Buildroot] [Hosting] Unplanned outage: Hypervisor issue with gprod1 on primary Ganeti cluster

[Peter Korsgaard]

>>>>> "Lance" == Lance Albertson <lance at osuosl.org> writes:

FYI,

If you had issues accessing sources.buildroot.{net,org} today, this was
most likely the reason.

The machine is up and running again.

 > All,
 > At approximately 2:36AM PDT (0900 UTC), one of the hypervisors (gprod1) in
 > our primary Ganeti cluster started having hardware issues. This took down
 > all of the instances running on that node. I attempted to bring the node
 > back online however the hardware issue prevented it to come back online. At
 > that point I failed all of the VM instances over to their secondary nodes
 > and forced another node to become the Ganeti master (since gprod1 WAS the
 > master). All of the instances were back online by around 7:40AM PDT (1400
 > UTC).

 > Everything at this point seems to be back to normal (except for gprod1). I
 > will look into bringing gprod1 back online later today.

 > Thank you and sorry for the outages this caused.

 > -- 
 > Lance Albertson
 > Director
 > Oregon State University | Open Source Lab

-- 
Bye, Peter Korsgaard