Arrgh...

[SB2]

from distributed.net

:: 18-Apr-2004 15:40 CDT (Sunday) ::
Ooops. Looks like we forgot to restore an incremental backup from just before
the crash. In order to restore that backup and have it take effect, all changes
participants have made since fritz went online well unfortunately be lost. We
will also have to re-run all stats for the past two months or so, which will
take a few days.

Sorry for the inconvenience.

[Bionic_Redneck]

I was wondering what was up with the stats and now I know

[em99010pepe]

I lost all my work. I have here 500 RC-52 blocks to send but I don't know what I should do.

[Bionic_Redneck]

you just have to wait until server comes back online.

[SB2]

a quick follow-up;

:: 18-Apr-2004 22:28 CDT (Sunday) ::
The restore is done and fritz is now churning through log files. I'm about to
turn apache back on. Something to keep in mind is that many user changes take
effect on the log date that the change was made. So if you retired an account
or joined a team on March 3rd, you won't see the change take effect until the
daily stats run for March 3rd.

Looks like the database is up Feb. 11th so far. Perhaps by tomorrow I'll be able to join the team.

[NeoGen]

I've just seen the stats on distributed.net. :shock:
But from what I understood from the posts previous to mine, it will be updated, right?

[em99010pepe]

I've just seen the stats on distributed.net. :shock:
But from what I understood from the posts previous to mine, it will be updated, right?

I hope so.

[SB2]

I've just seen the stats on distributed.net. :shock:
But from what I understood from the posts previous to mine, it will be updated, right?

The stats are now up to Feb 21st, so one can hope all is not lost, yet.

[SB2]

Statistics are now current.

Also finally recieved my password and should (fingers crossed) show in tonights stats run.

[Ototero]

More AAAARRRRRHHHHHHHHH

Thanks to poor driver support, we had been running for who knows how long with
3 failing drives in the raid10 array that housed the database. But that wasn't
actually what caused the outage... if a machine with an 8500 in it goes down
unexpectedly (think power failure), the controller can't trust the data on the
drives to be in-sync, so it needs to rebuild the array. Unfortunately, one
of the drives it picked to be authoritative was failing, and decided that it
wasn't going to give up it's data.

Unfortunately we've been unable to recover the array. We tried using spinrite
as a last resort, but at the rate it was going it would have taken something
like a week to recover the drive. This means that when we get back online,
we'll be running from a stats backup taken Nov. 6, about 4 days before the
failure. Any changes made to participant accounts or teams in the meantime will
have been lost.

In an ironic twist of fate, we've been working on getting a new machine in
production that would have allowed replicating user-modifiable tables (ie:
participant accounts and teams) to another machine. Had that been in place we
would have lost very little, if any, of this data.

The current situation is that we've bought 3 new drives and used them to
rebuild the array. We've also taken this opportunity to upgrade to FreeBSD 6.0.
But now any time we try to access the array, the machine reboots.

Once someone is on-site to investigate we'll hopefully know more.

What a waste of my crunching. I'm off to Folding@Home until the cows come home ;)