If you pay any attention to famous business case studies, you’ve probably heard of Pixar’s Toy Story 2 disaster. But if not, here’s the (toy) story:
The Loss & Recovery of Toy Story 2’s Data
Almost 20 years ago, Pixar was using Linux machines. On Linux machines, there’s a command that removes everything below the current directory on which it’s run. Someone inadvertently ran that command at the root level of the Toy Story 2 project.
Entire sequences started disappearing, right in front of everyone’s eyes. Oren Jacob, former Chief Technical Officer of Pixar, called systems to demand they unplug the machine immediately. After doing so, then plugging it back in a few hours later, they realised most of the film was gone. 90% of the movie had been deleted by the command.
It would have taken a full year of all 150 Toy Story 2 crew members to recreate all the work that was erased in roughly 20 seconds.
But Galyn Susman, Technical Director and now Producer for the Toy Story franchise, didn’t panic. The first thing she did was check their backups of the film. Pixar has always had a history of backing up its data.
Unfortunately, that’s when they realised their data backups for Toy Story 2 had failed for the last month.
Ideally, backups should be tested every few days or weeks, in order to make sure your data is all there. But at the time, Pixar was not in the habit of continuously testing their backups. The problem was that Pixar’s tape drives kept hitting the maximum file size, so no new data was being consistently written to the drive.
Luckily, Galyn had a backup of the film on her home computer. She had been working from home due to giving birth to her son shortly beforehand, so they had connected her machine to the local network and copied the whole file tree over. Her computer would then receive incremental updates over her ISDN internet connection.
They brought her computer to the studio, plugged it in, and were able to retrieve their data and footage. The backup was about two weeks old, but the crew was able to verify about 70,000 files, then checking 30,000 files by hand.
It was a miracle, but they were able to salvage Toy Story 2… only to scrap the entire film and recreate it within the 9 months before its official release date.
For further reading, Oren Jacob responded to a question on Quora with his full statement of what happened.
How Did Toy Story 2 Get Deleted?
In an editorial interview with Oren Jacob, he explained how this error would’ve been difficult to prevent:
“You have 400 people on the network and they all have to have like pretty massive access across the board to the whole project, so it’s hard to like, limit the damage. It could happen from almost any terminal. Pixar being a wide open Unix environment meant that it was very promiscuous. You could change directory…or walk across the network and log into Ed Catmull’s machine or Steve Job’s machine if you wanted to.”
The common way to prevent an accidental command like this is to restrict user permissions to only the files they need. But, because of the way a Pixar project like Toy Story 2 works, almost everyone needed permissions to read and write to the master machine.
What Toy Story 2 Teaches Us About Redundancy
The truth is, human error is a frequent cause of disaster. One wrong keystroke can result in significant data loss and downtime. Similarly, one wrong email open can let ransomware wreak havoc on all your files.
No matter your project type, a data backup strategy needs to be priority for your business. And as this Toy Story 2 case study illustrates, it’s not enough to back your data up to on-premise drives. Backing up your data to an off-premise server and running regular live backup tests ensures your data is verified, safe, and secure at all times. Backing up your data to multiple off-premise locations can add further layers of security to your data, in the event disaster was to strike one of your colocation sites.
If your business doesn’t have a business continuity plan or data backup protocol in place, contact us at Dynamic Business Technologies. We’ll make sure your data is safe and secure, backed up to hosted private data centers right here in NSW.
