As some of my friends on Facebook will know, I recently suffered a catastrophic data loss. The result of the failure is setting up an entirely new backup system for our family data. Now, before I describe the failure, please note – I consider myself an idiot for what happened. I set myself up for this, and it happened.
I am a data/media pack rat. I have been using Netflix for at least 10 years now, and every time I would get a disc, I would also make a copy of that disc, and rip the movie/TV shows into H.264 to put into iTunes as a home library. At the time of this writing, I have 1,711 movies, 6,535 TV shows, and my history as an audio junkie in high school and college means I have 10,852 songs, all ripped as Apple Lossless, taking up another 500MB or so. The song management has been very precise – I’ve added my own album artwork (embedded in the file) as well as Lyrics, custom genres, etc. In addition to this, I have about 1.5 TB of home movies that have been shot between my wife and I, dating back to before we met. Around 1TB of that is a capture of analog tapes as standard definition DV files, and standard definition files of Digital-8 tapes. The rest are Quicktime or MPG movies captured from digital cameras, and a large chunk of H.264 captured from iPhones and an HD Sony Camcorder.
My Life is Over
It’s this last 1.5TB that caused this whole mess. Like an idiot, I had this on a single hard drive. It was on an internal 1TB hard drive that I would plug into an external USB2 reader, and when that ran out of space, I moved everything to a 3TB Samsung USB3 drive.
1TB drive in external USB2 reader
I felt I was doing everything right with this drive. It spent most of its time in a drawer in my bedroom, and would only come out when I had a new batch of videos from our phones or camcorder to archive, or if I wanted to pull old data off the archive to build some project in Final Cut Pro.
However, as you can see, this was a single point of failure. This drive had no backup. And so, one day I working on the kitchen table, like I am wont to do, and had this drive out as I was working on some projects – a singing recital from my daughter, and a video blog I was thinking of making. The drive was plugged into the wall outlet on the floor next to the table, and plugged into my Macbook Air. And just then, my dog decided to say “hello, daddy!”. She lumbered over to me, pulling on the power cord, causing the external hard drive to crash to the floor.
I quickly unplugged the drive and placed it back on the table. Plugging it in, I crossed my fingers to see if I could read it and… no dice. It wouldn’t even mount. I… was… crushed.
After sitting in my own despair for a few hours, and kicking myself for not having any other backup, and thinking that I just may have lost my entire library of home movies, I got to work trying to rescue the drive.
Quick aside: I should also mention that the original 1TB of DV files of analog and digital tapes? I luckily had another backup of that. When I originally captured the tapes, I burned these DV files as data DVDs.
Data DVD Backup of Tape Based Camcorder
So I had, sitting in my closet, 130 or so data DVDs of all of this. However, pretty much everything after 2008, when we started shooting stuff on iPhones? Gone. There was no other backup for that, an no other backup for things I had shot with the H.264 camcorder, and all my Final Cut projects were lost.
I found a place in San Jose that does disaster recovery, called Drive Savers. They do clean room disassembly of the drive to check the platters and all that. Even though it was late on a Sunday afternoon, there was a person there – not just some random 800 number call service, but one of their engineers who walked me through everything they would do, sent me a pre-paid FedEx 1 day shipping label, and said that if they couldn’t recover anything, there would be no charge.
I shipped it off, and kept my fingers crossed. Unfortunately, there was nothing salvageable. The drive, apparently, didn’t have the kind of sensors that tracked motion so as to stop the drive spinning and move the heads to an idle position. The drive just kept spinning during its fall, the drive heads got misaligned, and the platters were all gouged. In case you are wondering – it was a Samsung USB3 drive. Here is the link, just so you know not to buy this f$#@!ing thing.
(Luckily) I Recover Everything
So, things were looking very, very, very dark. However, I still had that original 1TB drive from before I had moved things to this 3TB drive. Even though I had re-formatted that drive for future use, I had never used it. I was able to use Data Rescue 3 to read the drive. I wasn’t interested in the DV files – as I could get those from the data DVDs. But using this software, I was able to get all the digital camera files from 2006-2007, and all the iPhone files from 2008 – 2011 when the drive got full. Phew! I also remembered I had been keeping my 2012, 2013, and 2014 iPhone files on my Macbook Air for a long time, and I had been backing that up to a Time Machine disk at work. Going in the next day, I was able to recover all of those. WOW! So, things were looking up.
After spending a couple of weeks, I think I’ve recovered 99% of these files. The ones on digital camera that were in the 2006-2007 time frame were recovered, but unfortunately I don’t know exactly when these were shot. The file names were dates of the files, and the file names were lost during the reformat. All the iPhone files had been tagged with metadata as to when the ‘record’ button was pressed, so I know all those dates (I love you for that, Apple). And all the DV files had been named by their date on the data DVDs, so all was well.
However, I still needed a new backup solution. And by new, I mean “something besides not having one”. And we are talking a lot of data. There would be at least 1.5TB of home videos, so an online solution wasn’t going to be cheap, nor fast. I don’t like online solutions specifically for that reason. I’ve heard it taking up to a week to do an initial backup for much less data than that, and if I needed to recover, that would be at least another week, unless I wanted to dip into my pocket for a hard drive of the backup to be shipped to me.
Let’s Really Do Backup
A buddy of mine and I had decided long before this happened that we should use CrashPlan Friends, where we would back up to each other’s houses. I would back up my stuff to his house, him to my house. In the case of a catastrophic failure, it would just be a matter of picking the backup drive up from the other house. And at this point, I decided: I’m not just backing up my home videos…. I’m backing up every… friggin… thing. Movies, TV Shows, music, all of it.
And so… here we go. I decided to buy a Drobo 5N, a 5-bay, RAID technology NAS (network attached storage). I would initially fill it with 4 3TB drives in a single drive failure configuration, meaning I would have something like 8.5TB of storage available for backup. Looking at all my media, I think I have something like 5-6TB of data. I went with the Drobo because I can just slap another drive in without rebuilding it (it will rebuild on the fly), and I can even mix and match hard drive sizes. I priced the solution both on Amazon and Newegg, and Newegg came in cheaper. So, $1K later, I was off!
UPS Has Arrived!
Shortly after burning a hole in the credit card to get the equipment, it arrived.
Unboxing… The Drobo and the 4 hard drives.
The Drobo comes in a cute little carry bag. Not sure exactly how useful that is.
Even more unboxing…
Just about ready to go. Drobo seems to have tried to go the Apple route with its packaging. Very minimalistic, and even has a sticker with their logo, just like Apple tends to do.
One of the four hard drives. It is a Seagate Barracuda 3TB drive.
The drives are all mounted. Let’s get going!
Configure and Go!
After installing all of the drives and plugging it into power and the home network, it was time to fire up the Drobo “Dashboard”, an application for Windows/Mac that allows you to manage the Drobo.
Where are you, Drobo?
And this is where some problems started. The software couldn’t find it. After a lot of investigation, I found that my Ethernet hub, while powered, isn’t working. I have no idea how long this has been the case, but it could have been for a year or two. My whole house is wired for Ethernet, but apparently none of the rooms have anything plugged into it. This is stunning to me as just 10 years ago I could be found saying that I wouldn’t trust wireless internet as far as I could throw it, but clearly everything in the house is running wirelessly now. So, I plugged the Drobo straight into my router, and tried again. Unfortunately, I still had some problems.
I can’t find you…
The Drobo dashboard software is a little finicky. It would give up quickly when looking for Drobos, and I would occasionally see the above screen. During configuration (as I will describe below), occasionally the Drobo has to be restarted. And this could take some time, at which point this Dashboard software would freak out and say it couldn’t find a Drobo. Eventually, though, it would find it, and the screen below would appear.
A-ha! There you are!
After about 20 minutes, the Drobo would fully configure itself. As I was looking at the lights on the Drobo, I would see it discovering drives, formatting them, and configuring them for use. Eventually, I got this screen, indicating I had installed 4 3TB drives, and they were ready to go.
Checking over to the “capacity” screen, I saw that I had 8.08TB available.
Me John, Big Tree
That’s 4 drives in a single fail over setup. Which means, you essentially lose one drive’s worth of space so that you can reconstruct data if any of the drives fails. So 3*3TB = 9TB, but after formatting, that leaves about 8TB of space left. The capacity screen below shows how Drobo uses the disks.
The foot bone’s connected to the… ankle bone.
Now, the way I wanted to use the Drobo is as a peer on CrashPlan. What this means is that I would do a CrashPlan backup to a friend’s house, and they could do one to mine. Typically, this means having the free CrashPlan installed on a PC/Mac at the remote location, with the backup drive attached to that machine. Given that the Drobo itself is the machine, this means you would install the CrashPlan client on the Drobo. Drobo has a set of apps you can install on it, and they are under the Drobo “Apps” section. Clicking over to that, I found the CrashPlan app.
Pick me! Pick me!
Clicking on it, it went to install the app.
Give me a minute…
And… uh oh… problems.
You followed the instructions, but no dice!
It turns out Drobo has updated its firmware, making the CrashPlan client install unusable. And this is where I found out that CrashPlan doesn’t really support this (WTF, it sure seemed like it did), and Drobo doesn’t seem very interested in support it, either. There are ways to make it work by installing custom apps. This was not something I was looking forward to doing, but I went down that rabbit hole.
At the end of the day, I ended up with two additional apps running on the Drobo, which the dashboard didn’t quite know what to do with – it knew it was running, but didn’t know the status.
Are you hacking me?!?!
But, at least I was able to install Crashplan now.
All our CrashPlan Belong to You
Unfortunately, this still didn’t make things work very well. The “ssh” client that I had to install decided it didn’t want to SSH anymore, even though it said it was running. And CrashPlan needs you to make some changes via SSH to make this work.
At this point, I was, shall we say, pissed.
So, I have instead decided to abandon this approach. It doesn’t mean I’m abandoning CrashPlan, but what my friend and I are going to do instead is share drives from our local network (under password protection) with each other. By doing it this way, I can mount the remote drive, and then just use CrashPlan’s “backup to external hard disk” option. I could also decide to use Apple’s Time Machine if I wanted, or to just flat out copy files to the remote device – don’t run it as a backup at all. I’m strongly considering this, but given that manual backups tend to mean “didn’t back up”, I first decided to stick with using CrashPlan.
But before I send the Drobo to my friend, I want to do the backup locally. This will be much faster as the home network is going to be faster than a remote network connection. So, I went into the CrashPlan client, selected all my sources, and selected the Drobo. It turns out I have 5.5GB of data. And, unfortunately, even on an internal network, CrashPlan says it will take, get this, 2 months to back up. Egads!
This is gonna take a while…
So, at this point, I’m reconsidering some of my options. I still want to backup everything, but I probably don’t need to have the backup be a “backup”. For example, once I put a movie in my library, the odds that it will change is… slim to none. Since I’m manually copying the movie over to the library, I can also manually copy it over to the Drobo. That is more work on my part and always a chance I could miss something, but hey, I’m a programmer… I can write some scripts to see if things have gotten out of sync.
Back up your sh*t! Serously… back it up. Not just because it might be important to you if you run a business, but because you just can’t get back lost memories. You would think I would have learned my lesson when our family went to Barcelona when the kids were little, and our apartment was robbed. They took the laptops where I had been downloading all my digital photos. This was before the days of cloud photo backup, so nearly that entire vacation is lost. Yet, obviously, I didn’t learn. And here I was again, about to lose all the video from our children’s lives!
Don’t be cheap. Back up your sh*t. While it might be trite to say it… it’s only money. Memories are priceless.