Pixar’s Toy Story 2 was deleted twice. Each instance, they had to rebuild the movie.
Two months and hundreds of man-hours of work vanished
It all innocently started with a fix to Woody’s hat. Woody needed a slight wardrobe upgrade—that’s all. Just a hat reposition. While one of the editors was tinkering in the file system, trying to install a new and improved Woody, something much unexpected happened.
THE folder containing the Toy Story 2 movie—that at one point had 40 files— suddenly had four then none. What Oren Jacob, former Chief Technology Officer (CTO) for Pixar recounted was an entire movie deleted off the company’s servers.
Pixar was only able recover parts of the vanished movie
After a streak of luck after a tens of thousands of dollars, hundreds of man hours and some bright engineers, Pixar was able to recover parts of the movie. With much relief to the producers, there was hope to still complete the movie on time, though over budget. The silver lining is that the 150 hard work by dedicated animators with drive and commitment to quality movies had been recovered!
Don’t worry- the story thickens. Ever heard of ‘/bin/rm-r –f*’ ?
A few weeks after the first recovery had revived the Pixar movie, Jacob was looking through the directory where all the assets for Woody were stored. What he noticed, when refreshing he noticed less and less files in Woody’s folder. Eventually, seeing this exact message:
“Directory no longer valid”.
The reason for the error—the folder no longer existed.
Moving up in the directory, he saw Hamm, Potato Head and Rex (other characters in the movie) vanishing in plain sight.
The command that Jacob had accidentally typed: ‘rm –r –f*’. In plain English, this command tells the system to remove every file within the current directory. What this meant for Pixar’s Toy Story 2? Everything was deleted—nothing remained.
At that point, panic struck in for the Toy Story team—someone had even given instructions to yank the power cord and network connection of the server holding the movie’s files. When the machine was rebooted, only 10% of files remained. That means nearly 90% of the movie had been deleted!
Normally, deletions wouldn’t be a problem. In fact, for Pixar (and most businesses actually) files get erroneously deleted all the time. What Pixar expect—as your business should as well—were backups of the movie that actually worked. Unfortunately for Pixar, their tape backups were corrupted. These backups were NOT continuously tested—leaving the company with a bunch of incomplete tapes. The backups could not get the Toy Story production back on track.
Lesson learned: backups are only as good as their testing. With no testing, who knows if anything actually got backed up (Note: Dynamic Edge performs DAILY TESTED backups of your systems to avoid a Pixar devastation).
On this National Backups Day, I want to think about what should you be doing to be safe…
The general IT golden rule is always the rule of 3—to lower your risk of a karmic event, always have 3 copies of your important data. The original, a backup of the original and an archive of the backup. Each of these backups should be kept in different locations. If you’re using drives, that means a lot of realty—especially if you’re backing up a lot of data!
You should be replacing backup drives at least every other year. That means someone not only needs to be responsible for swapping drives daily, but someone has to make sure to change out drives in the rotation often! And then keeping these backups in a safe place for that just in case instance where you need to recover that really important file.
Isn’t there anything better?
While disk backups were the standard for the last decade, our cloud technologies have become much more reliable than any disk backup process. Our cloud has built in triplicate replication and eliminates the need for someone to manually swap out disks.
But benefits of cloud backups just keep coming:
Get you back online faster— cloud backups are continually working—and if a problem exists, you are back online within a couple hours. Rather, to restore from disk, you would need to first get access to the backup drives, then transport it to the office, then start the restore process. That means hours of additional wait time!
Backups that last longer— as we mentioned, your disk backups only last about two years. BUT cloud backups last in duplicate or triplicate forever (well as long as you pay for the space J) and you don’t have to worry about the backups failing- they are constantly being tested and maintained. That means that even if you have to restore from a backup, you have it at your fingertips with no worries or what if’s.
Backups that cost less—costs associated with disk backups are multi-fold. Think of cost of storage space, labor costs to regularly replace drives, cost of the drives themselves. With cloud, you just pay for the space you use—no more!
I’m worried about your networks, your data, your systems. I think about them day in and day out. I think about the ‘what if’s’. If something were to happen, is everything safe. The reason why I’ve moved my company’s backups completely to the fitCloud and encourage all of my clients to use our cloud solution is that we know it works better—safer—faster – and cheaper than any disk or physical backup. And on top of all of that, you are working directly with us—people that have been and continue to be dedicated to getting your business the right technology to fit your needs.
A final note on Pixar’s Toy Story: An employee had a copy of the movie on her home computer. After another grueling week of non-stop recovery (sleeping bags, coffee and pizza-filled nights and days), the team was able to recreate the movie as best they could. The entire 100 Million dollar project weighed on one computer.
Doing some backups soul searching? Confident that your systems are backed up AND tested? Are you sure your tapes are even working? Contact us TODAY to learn how to set up proper backups for your data.
