Seti@Home Outage

[Scavenger7]

February 23, 2005
Update: a breaker blew and the power for the entire lab was off for several hours. Power returned at 23:30 UTC, but we will be dealing with fallout for a while - the project will probably be down all night and through tomorrow morning.

February 23, 2005 - 23:30 UTC
A sudden, unexpected power outage due to a blown breaker shut the whole BOINC project down for several hours (along with all the other projects in the lab). The cause is still unknown (which is scary), so there will be a scheduled power outage in the near future to hunt for electrical problems. We do know this: we just can't seem to catch a break around here.
We were able to gracefully shut down many servers on battery backup (UPS) before the batteries drained, but not all of them, including the new BOINC database server. So the data is scrambled, and mysql refuses to start. Our last backup to tape is a week old. This week's tape backup was about 60% finished when the power went out (Murphy's law in a nutshell).

The good news is we have a replica database which should be up to date. The bad news is that this had disk errors upon booting up and its drives are still resync'ing. After that, we'll have to check the table integrity on the replica - if we're lucky and mysql is able to start, we can then dump the data from the replica back onto the master and continue right where we left off.

Earlier this morning the project was off for some routine maintenance (tweaking the BIOS on the database server to get rid of spurious error messages and snapshotting for database backups). An hour after we brought everything back up the power went off.

http://setiweb.ssl.berkeley.edu/tech_news.php

[Lord AnthraX]

Man I really hope they are able to fully recover soon. I've just now been getting a few machines to run it to help contribute what I could to SETI.

P.S. Can I have 3 or 4 computers use the same username to send/receive data?

P.P.S. I've been using the classic Seti@Home program, would that still work or do I need to install BOINC?

[SirFrank]

P.S. Can I have 3 or 4 computers use the same username to send/receive data?
Yes! I have 3 computers running Seti myself. You just use the email and handle you signed on with.
P.P.S. I've been using the classic Seti@Home program, would that still work or do I need to install BOINC?
You do not need to install BOINC for now anyway. Dont know if they plan on changing it.

Scavenger7!
Thanks for the update. I was wondering what happened.:t

[Scavenger7]

Originally posted by Lord AnthraX

P.P.S. I've been using the classic Seti@Home program, would that still work or do I need to install BOINC?


They are in the process of switching to boinc. once that is done classic will end.
Until then you can still use classic, they will email everyone when it is time to change.


EDIT:

From the front page of seti classic.

NOTICE: SETI@home is in the process of switching to new software called BOINC, which lets you run other projects (like Climateprediction.net and Einstein@home) on your computer as well as SETI@home.

http://setiathome.berkeley.edu/

[Lord AnthraX]

Also, which link do I use to connect to the seti project.

[Scavenger7]

It was worse than they feared.. :(

February 24, 2005 - 23:30 UTC
Update on yesterday's outage: We are still dealing with some database fallout. Most of the classic SETI@home systems are up - enough that we can serve workunits to users. However, BOINC is dead in the water until we get at least one database server up and running.
With the master database corrupted beyond repair, we turned all our attention to the replica. Its disks finished sync'ing last night, and after some file system checks the machine booted and mysql started just fine. A battery of tests revealed no corruption.. until we got to the result table. Of course, that's by far the biggest and most important table in the database. We are attempting to repair it now.

Assuming we can repair it with little or no data loss, we will then dump all the data from the replica back onto the master. If we're lucky, this will be done by tomorrow morning and we can start revving all the engines back up.

Please note that since it was a slower machine than the master, the data on the replica database server was about 30 minutes behind real time. We did try to limp both systems along to sync the replica data up even further but no dice. So, when we do get back on line it will be as if there was a half-hour hole in time during which all uploaded results were lost (and any user profile updates, message board postings, etc.). We sincerely apologize to all our users for this loss.

http://setiweb.ssl.berkeley.edu/tech_news.php

[j.m@talk]

Not very good at this stuff are they :p

"It wouldn't of happened in my day" ;)


I guess thats why I'm up to my neck in Wu's ......... I gottem' comming outta my ears nearly :x

:t