I was a programmer at the time, and it was my week to be on call. I received a "panic" call from the system operator one night. He told me all of the jobs were "bombing." I dialed in to the system to check it out. When I finally got connected and checked out the problem, I found our production file library was missing. It was completely gone.
When I questioned the systems operator, she didn't know anything about it. She said everything was going fine, and then, all of a sudden everything was erroring out. After looking at the logs from the console (interactive) job used by the operator, I found she had gone into a menu that allowed for environment maintenance.
At the time, we were using the MAPICS system and a product that allowed us to create multiple instances of the MAPICS environments. We had environments set up for production/testing and multiple different business units, etc. She got into this menu that allowed her to take an option that deleted our main production environment and all of its associated libraries. When I questioned her again, she said she couldn't have done something like that. She always went by the "run sheets".
After coming into work now and making a few other phone calls to notify managers, I started to restore and rerun all of the day's updates, again.
The problem, it seems, stemmed from the operations department run sheets. All of the run sheets were "sequenced" from their original sign-on menu. For a particular job, (for example) they knew they would take option 4, 6, 1, 1, 1, and press enter three times to run a particular job. They had every job outlined this way. They had gotten away from writing the "menu descriptions" on the run sheets. This all worked fine -- if they started the process from their main menu. In this particular case, systems operator forgot to go back to the main menu and completed the sequence for the next job -- with disastrous results.
It took us about a day and a half to set things straight by the time we were done. Needless to say, this was a major blunder!Where do you want to go next?