Why You Really Should Have a Backup

I had a bit of a crazy day today with this website, so this is going to be my slightly unconventional post for the week. As you could tell from the title, it's about why you should really have a backup. From personal experience. (And next week, perhaps I'll talk about how you can make a backup if you don't have one.)

I worked on my website a bit this morning; I wanted to post a couple of new things and make some updates to my homepage. As part of this process, I needed to upload a couple of files, which required that I open a terminal window to access the server. No big deal, I do it all the time.

Another thing I needed to do today was pick up in my room/office area. When I took a break from working on my website, I was picking up. I needed to move a desk; the path of the leg was right in the way of the computer, so I needed to turn off the computer to move it out of the way. So I flipped on the monitor, noticed there were two terminal windows open ready for a command, and typed sudo halt in one of them—the command to power down the system.

Then I noticed I'd run it on the server.

You see, Amazon EC2 has this thing where some of your data isn't guaranteed to stay on the server when it powers off.

I rebooted the server from the management console, but the damage was done—all my posts since May 5th were gone. I panicked, of course, and then I had to go eat lunch. When I got back, I fumbled around in my backups folder and found one from June 9th. Which should have been good, because I'd written all the following blog posts before I left to go camping on the 12th. But after I restored it, there were still five posts missing. Apparently the backup software doesn't back up posts that are not yet published publicly.

Amazingly, I was able to recover every word of my posts from Google's cache of my website. (The pictures are missing from the cache, but the links and filenames aren't, and I still had them on my hard drive.) But searching the internet for your lost data isn't a very good backup plan. The whole thing wasted me half an afternoon.

And I got very, very lucky.

Only a week ago I overwrote three hours of work by accidentally typing ">" instead of ">>" in a script. It's amazing how the stupidest user errors can make the worst crashes.

I could have been up and running again in roughly five minutes had I kept a backup that covered everything. Which is why I now have a weekly backup that runs right after I update my website every week. You should too.