Wednesday evening my server (webwizardry.net) went down due to hardware failure. Near as I can tell, the power supply went bad, and the unstable power fried the main hard drive. One of the NICs was also damaged. Before the hard drive failed completely, I was able to retrieve most of the data, but not all of it.

This has been a difficult couple of days. For the first 24 hours, I didn't even realize that the power supply was the problem. I wasn't able to recover the whole operating system, so I had to reinstall from scratch, then rebuild what I could from backups. At this point everything seems to be running smoothly, although there's still a lot left to work on, and there's no guarantee it won't suddenly become unstable again. However, assuming all the faulty hardware has now been replaced, the system will soon be back up and running even better than before (along with the reinstall came an OS upgrade). You can bet I'll also be implementing a regular scheduled backup procedure!

This is the second time in the server's four-year history that I've actually lost data, and the first time was just some MP3s stored on a software RAID-0 across hard drives that were pulled out of a trash can marked “BAD” - so that was sort of expected. It's also the first time I've ever had serious downtime. While I was moving from Arizona to Oregon, the server was down, but I made arrangements for everyone's home pages and e-mail to be hosted elsewhere during the move, and that was all planned out ahead of time. I think this is the first time in four years I've had an outage that lasted more than 8 hours. This may sound silly, but I'm a little depressed about the whole thing. Maintaining a stable server for so long is something I've taken pride in.

If you sent e-mail to me, or anyone else who uses my server, between Wednesday 11/20 and Friday 11/22, please re-send it.

A few random statistics: the average number of e-mail messages my server processes per month has increased from under 8,000 to over 11,000 over the past six months. The number of HTTP hits has fluctuated between 16,000 and 23,000 per month. Over the past four years, every piece of hardware has been replaced at least twice, some things three or four or five times. 8.1 is the fourth version of Slackware to be installed.