Disaster Recovery Guide - True Stories

This is a growing catalog of real events that led to loss of availability or other situations that might have required invokation of the disaster recovery plan. If you have a true story to add to this collection, please send it to us via the feedback page.


1) THE INSIDER JOB
The first story demonstrates how seemingly little things can have serious consequences:

Over a period of nine months, the number of computer malfunctions within a large company had risen from an average of two per year to critical levels. The impact was such that the business fell behind with its invoicing systems and had to buy processing and backup from third parties. As it could not deliver some of its services reliably, it started to lose the confidence of its customers. The situation began to spiral.

Eventually, the company suspected foul play may be involved and called the police. Secret surveillance equipment was installed to monitor staff. One was filmed lightly scratching circuit boards in disk units and also attaching paper clips to them. Both these actions led to a short circuit.

When confronted, he confessed everything. His motive was to earn overtime, which was required to process the overlap work which was delayed by the malfunction. He netted 689 UKP over the 9 months. The company lost at least 500,000 UKP.


2) CONFIDENTIAL LITTER
When an explosion occurred at the head office of a major bank, the surrounding streets were littered with papers containing confidential customer information.

As well as recovery from the physical inclident itself, which happened on a non-working day, the bank also had to contend with an issue of breach of confidentiality and the resultant publicity which followed.

This was possibly the one area they had not covered properly in what was otherwise an exemplary disaster recovery plan.


3) THE INSIDER JOB 2
A series of incidents occurred during one night shift at a major computer installation. A series of power-downs prevented further output as there was no engineer present to re-initialize the system. Shift staff consequently had to be sent home each time.

The cause was eventually discovered. An operator was so jealous of his unfaithful girlfriend (allegedly!) that he discovered a way of "checking up on her". On random occasions he would turn the mains switch off and then back on again, and would subsequently be sent home... unexpectedly. The idea occurred to him following a genuine failure.


Source : http://www.disaster-recovery-guide.com/bcpg.htm