Seti@Home Outage!

**Scavenger7** · 02-28-2005, 04:12 PM

February 28, 2005
Around 18:00 UTC we had another unexpected lab-wide power outage. Systems were able to shut down more gracefully than last time, but we are leaving many services off as we survey the damage. The cause of these outages is still unknown.

http://setiweb.ssl.berkeley.edu/

**Scavenger7** · 02-28-2005, 11:04 PM

February 28, 2005 - 22:30 UTC
So we had another unexpected lab-wide power outage again this morning. This time around we had the BOINC database on battery backup so we were able to shut it down safely. After the power returned we brought the database back up briefly to check it out - and it's in perfect health. You can all thank Court for bringing in his personal UPS (and leaving his own systems unprotected) to put on the BOINC database server until we were able to obtain a new one.
But we shut the BOINC database right back down, and will leave most of the BOINC back-end services off for the time being until we have all our important systems on smart UPS (the systems will shut themselves off once they realize they are on battery power). This has always been the future plan (and please note that our previous configuration allowed for zero or minimal loss in the event of a power failure), but now that frequent random outages are part of the scenario, it would make life easier not to have to do damage control every time.

We are actually going to take this time off to do additional maintenance. For example, the disk array holding the upload/download directories is 98% full - Jeff discovered a bug in the file_deleter code that left a lot of old workunits around. So we need to get rid of those stale files before anything else.

http://setiweb.ssl.berkeley.edu/tech_news.php

**j.m@talk** · 03-01-2005, 03:55 PM

Yer my sistem is backing up wiv seti files again ........ Still there yas are....... & my Einstien wu's are being looked at eventually ......... It is a hulabloo

**michaeln** · 03-01-2005, 06:07 PM

Here's the latest:-

March 1, 2005
Power Outage Update: Since the cause of the random power outages is still unknown, we are leaving the data server off during the evenings (users will not get workunits/send results at that time). We can handle power outages during the day while we're at the lab, and are working towards a better system to handle outages at night. Meanwhile, campus is trying to diagnose and fix the problem (which effects the entire building - not just us).

**j.m@talk** · 03-01-2005, 06:13 PM

They needs a huge diesel generator that goes thud thud all day & all night ........... The residents will complain so much that the electric company will have to sort it "Pronto"

**porsch1909** · 03-02-2005, 03:51 PM

Still no working

**michaeln** · 03-02-2005, 04:42 PM

Originally posted by porsch1909
Still no working

Yup! I have a backlog of WU results to upload also.

**j.m@talk** · 03-02-2005, 05:35 PM

I got just over 2 Wu's left then I'm gonna be out of stock

**Scavenger7** · 03-02-2005, 10:49 PM

March 2, 2005 - 19:00 UTC
The building power is still untrustworthy. A diagnostic power outage is going to be scheduled for some time next week.
To clarify our current situation, all of our servers are in fact on UPSs and we suffered no database damage from the power outage this past Monday. What we do not have in place yet is a graceful shutdown system should the power fail and we are not here. We have installed the software on the servers that will enable them to recognize when they are on battery backup. We are waiting on the special communication cables that are necessary to connect the UPSs to the servers. They had to be special ordered and we expect them tomorrow.

While we have been down these last 2 days, we have been doing various maintenance tasks. Currently we are running a database backup. Once that is done, we plan to bring the project up for half a work day or so today. We will shut it down again at 01:00 UTC.

The classic SETI@Home project is currently up (but will also be shut down at 01:00 UTC.)

**j.m@talk** · 03-02-2005, 10:50 PM

Ya call that news

**j.m@talk** · 03-03-2005, 05:39 PM

I dunno how, but my mach has got 8 Wu's from somewhere & is setting about a crunchin' em ........ Still got 9 completed, "Ready to reports" on the shelves too .......... Ahh well at least its doin' summink

Mach "B" is crunchin' away too....... I dunno whats what wiv mach b....... It does its own thing & doesn't tell me

**j.m@talk** · 03-03-2005, 05:46 PM

March 3, 2005 - 17:30 UTC
The project is currently up. If the UPS communication cables arrive today we will have an outage to test the graceful shutdown procedures. If that goes well, we will bring the project back up and keep it up.

Well this is pretty conclusive

**porsch1909** · 03-03-2005, 05:55 PM

i thought you just said the project is up???

The project is currently up.

**j.m@talk** · 03-03-2005, 06:05 PM

Thats what they said ............ It must of bombed just after everyone went home

Those cables they were talkin bout are really expensive

I bought mine from a dodgy dealer in Texas & shipped it in for a 10th of the price APC wanted

**j.m@talk** · 03-04-2005, 01:43 PM

March 3, 2005 - 23:30 UTC
The UPS communication cables arrived and we spent a fair amount of time trying to get the UPSes to work. No dice. We tried everything (even going so far as to beep out the cables to make sure the pinouts were correct). Since it was wasting too much time we bailed and restarted the project for now. We'll likely shut it down for the evening again in a few hours.

Wodda bunch O' weiners

I bet they bought standard serial cables & are now realising that APC have out classed them ........... Ha ha ha ha

Thread: Seti@Home Outage!

Thread Tools

Search Thread

Display

Seti@Home Outage!

Update!

Posting Permissions