System i/iSeries users share interesting blunders

We've all made mistakes at one time or another and had to answer for our actions. Chances are, once you've made a major blunder, you don't make the same mistake twice. Here's our collection of the "whopper of all whoops" for your reading pleasure.

We've all made mistakes at one time or another and had to answer for our actions. Chances are, once you've made a major blunder, you don't make the same mistake twice. Here's our collection of the "iSeries blunders" for your reading pleasure.

Do you have a blunder that could top our 2006-2007 charts? Care to share your biggest goof-up of the year? Don't be shy. (I promise to use only first names.) Send in your blunder!

Click over to Search400.com's Hall of Shame to view the winning "whoppers of all whoops!"

*****************************************************************************************

His guilt was embossed on his forehead

December 2006 Winner

Greg writes:

Ages ago, in the days of the System 34, the programmers didn't have their own terminals. When we needed to edit, compile or test a program we had to use a terminal in one of the user departments. One summer afternoon, I was waiting for my program to compile (interactively of course) and I was feeling really, really, sleepy. My eyelids were drooping and my brain says to me "Look, you know this program takes at least 20 mins to compile and you can't leave until it compiles so you may as well go to sleep while it's compiling". "But" I say back "How will I know when to wake up?" "That's easy" says my brain "Just rest your head on the keyboard and when the compile finishes, the keyboard will start clicking".

I thought that sounded good so that is what I did. A few minutes latter I was woken up by the clicking. I logged off and returned to the IT department via the gents. As I walked in, the CEO was on his way out and he stopped and asked me how I was. "Fine" I said "and you", "fine" he said with a smirk on his face. I continued on thinking how weird he was behaving until I stood in front of the mirror and saw that my forehead had the pattern of a 5250 keyboard still embossed in it.

Lucky for me I was a good programmer!

*****************************************************************************************

Speed training turns sour

Kevin writes:

One of my first jobs with an AS/400 was as a data entry clerk/operator about 10 years ago. At some point along the way, our company's programmer came up with the fantastic idea of speeding up the training process by making a load menu with all the options clerks/operators would need to do their jobs. I forget the exact numbers, but I think option 7 was PWRDWNSYS and option 8 was timecard entry. Well, I'm sure you can guess what happened, but one day right in the middle of month end for accounting I went to enter time cards. That day my fingers moved faster then my mind, and 30 seconds later, the system took a dive, and sent four accountants who had been in "mid modify of data" into a panic.

Of course, the PWRDWNSYS mid day was bad enough, but this was an older system. 30-60 minutes to cycle when everything ended well... this day in the middle of month end closings, we were down for between 60 and 90 minutes, and boy was I embarrassed. I certainly never made that mistake again.

*****************************************************************************************

Who knew power strips were color-coded?

Susan writes:

Our computer room, as well as Programmer and IT tech offices are behind locked doors. As a result, the cleaning crew does not vacuum at night. I am not a neat freak, by any means, but one day the dirt and debris got to me. I went into the storage closet next door to the computer room, and brought out the vacuum cleaner, looked around for a plug, spied a power strip with an empty socket and plugged in. As soon as I turned on the vacuum, the phones started ringing. Not only had I taken down the AS/400 but our entire network. Who knew power strips were color-coded? Never, ever plug into an orange power strip!

*****************************************************************************************

New student grant field causes possible parental coronaries

Leigh writes:

My story is from the mid eighties, when I was working as an A/P on a System/36 running student grants. It was decided that the system would need to hold a brand-new field -- Parental Contribution. So I wrote all the new software, and I allocated an area of the file to hold the new data. Trouble was, this area previously contained spaces, and the Parental Contribution field was numeric. So you can probably guess what happened when the system went live -- hundreds of parents were informed, via a meticulously-tested letter, that their contribution to their child's education was to be $40,404.40. I never did find out how many of them suffered coronaries!

*****************************************************************************************

Was I the reason for special values that start with an asterix?

Roy writes:

This is a stupid error that I made over 20 years ago, and I was so traumatized that I'm only now able to tell about it. I was working as a relatively new programmer on a System/34. I was working on a report to compare the margins of salespeople who sold only widgets to those who sold no widgets. So I sorted out two files, one called NONE and the other called ALL, and wrote a report comparing the two. My supervisor loved it. When I was finished, I deleted the two files. DELETE ALL. Oops. The delete took a long time -- I had deleted more than half of the files on disk before I realized what I had done. We had to do a full restore in the middle of the day. My supervisor didn't love that! I've always wondered if my blunder was the reason IBM later came up with the idea of special values that start with an asterix.

*****************************************************************************************

Quick, hit the big red button on the wall

Dave writes:

A couple years back, I was sitting in my office working away on my terminal when I heard a strange noise coming from across the hall. After a couple of seconds I realized that it was HALON/Fire alarm in the computer room. I bolted from my chair and almost knocking several people over raced to the computer room. Kicking open the door, I saw two maintenance technicians sitting on the floor, in front of the air conditioner, with their blow torches trying to fix a leak in the unit. Knowing it wasn't a fire and that if we lost our HALON we could not get it refilled, I swung around and hit the big red button on the wall to stop the HALON dump. After holding in the button, I realized that the alarm was still sounding -- but all the computers had shut down. I raced over to the other side of the room to where the real stop button was and held it in until the computer room manager came in and told me that they had disconnected everything before the work started (except the alarm).

The little misadventure had our hospital down for about two hours and cost me a round of drinks for the IT staff to never speak of this again. Now they tell me before anyone is working in the computer room.

*****************************************************************************************

Disk space, give me disk space

Rudolf writes:

Although having much experience as a programmer/analyst on 32, 34 and 36, back in 1989 I was in the process of starting to learn the 400. On one of the courses that were setup especially for experienced programmers, they taught us the difference in commands between the 36 and the 400, etc. As one of the exercises I created a physical file with a file size of 1,000 records and looked at the disk space it occupied: only 4k! Which was normal because unlike the 36 where disk storage was allocated, on the 400 it was not. Wondering how that could be I tried the same with 1 million records. The result was still 4k. Then I tried with the maximum number of records, 999 extensions of the maximum extension size and the result was still 4k. Looking into the parameters of the create physical file command I then found the ALLOCATE (*YES) parameter. Forgetting to reset the number of records, the extensions and the extension size I reran the CL. It resulted in the 400 trying to allocate more than 1,000% of the total disk space. The machine was blocked for 2.5 hours before the sysop was able to cancel my job. Unfortunately for them, the machine was used not only for these courses, but also for their own administration and for customers to prepare and test their software. This all happed toward the end of the morning. Needless to say, we had an extended lunch break that day.

*****************************************************************************************

Who has time for testing?

Larry writes:

I had to quickly write a program to correct posting errors to an inventory transaction file. I used RPG with embedded SQL, selecting the data with an SQL query, and then updating the files natively. I wrote the SQL query and tested it interactively. No problem. Setting up test files and database overrides would take time, though, and I was in a hurry, so I decided to just single-step through the first couple of records in debug to test my program logic. When everything looked right, I just crossed my fingers and ran it. WHAM! I had a program error: receiver too small for result. The inventory balance for an item, which should have been in the low thousands, had tried to pass 99,999.

A little checking revealed that dozens of duplicate entries had been written to the transaction file. The program was looping, but I couldn't tell how. I double- and triple-checked my logic. I rechecked the query. Nothing seemed wrong, but when I ran a test (NOW I had a test environment) it kept looping. I set a statement to catch the last record in the file, and made that a debug breakpoint. When I single-stepped from that point in debug to the end of the program, it finally registered that I had typed EVAL *INLR = *OFF as the last line. In my haste to fix my initial problem, I had made the bonehead error of all time! In RPG, one normally sets *INLR (Last Record) to *ON to end a program. If one has *INLR = *OFF instead, the program will go back to the first executable statement and start over, forever. I didn't even get a warning from the compiler. I'm just glad the balance was only a five digit number.

*****************************************************************************************

Solid test environment turns soft

Phil writes:

A few years back I was hired as a consultant for a company using the AS/400 with J.D. Edwards World. I was told that they had a very solid test environment and to do all of my coding and testing there. Well, one of their logicals over a major JDE file, the F0911, was actually in test but was based on a Production file. So I deleted all kinds of records thinking it was just in test. Wrong. I had to stay there till past 11PM restoring those records!

*****************************************************************************************

To test or not to test

Pascal writes:

I was trying to promote objects on the test machine. Test machine maybe, but also used by international clients for their own testing. Fighting to get files unlocked, I ended up trying to kill so many jobs and subsystems the system stalled. I ended up powering it down with restart but this did not restart the web sites properly, and clients were unable to test until the next morning our time. I was not very popular with operations that day.

*****************************************************************************************

Y2K testing gone array

Michael writes:

I was working on a model F AS/400 when I first started as an operator/programmer. It was about 1995 and in discussions about Y2K, whether we were ready for it or not. We decided to move the system clock and time up to Dec 31, 1999 11:59:00 PM. We watched the clock change and the system seemed to be running without missing a beat. We saw the processor shoot up to 100%, and it was churning away as we tested a few programs and all seemed well ... until we looked at the active jobs and saw every job that was scheduled on the WRKJOBSCDE was running all at once.

Seems that the system said, "Oops… I haven't run the trial balance like I was supposed to 5 years ago, along with the month end processing, year end processing, nightly jobs etc…" We had some fun getting the system back into check since all the accounting periods were now closed and things were posting with dates of 01/01/00. Needless to say that concluded our Y2K test for the night, let alone the year!

*****************************************************************************************

Protecting PWRDWNSYS at all cost

Pascal writes:

A colleague of mine had tried, following my suggestion, to protect PWRDWNSYS. He obviously did not do it well as he came to me in a panicked state and demanded I power the system down immediately with restart. He had typed PWRDWNSYS and pressed enter, fully expecting to be told "not authorized."

*****************************************************************************************

Would you like fries with that?

Tom writes:

We still talk about the time we "cooked" our AS/400 systems. We had two of them, probably B or E models, with 10-12 towers of DASD attached to both. We had just installed a new alarm system to monitor the computer room, and management was still debating who should be on the alarm call list. Sometime over a weekend, the computer room air conditioner stopped working. And since there was no call list for the new alarm system, no one was notified.

We still don't know how long we cooked those systems. All we know is that when the first shift Operator arrived in the building Monday morning, he could feel the heat from the elevator lobby. The room was somewhere over 120 degrees (120 was as high as our thermometers could read). The system cabinets were too hot to lay your hand on. Any metal parts inside the cabinets would literally burn you if you touched them. Incredibly, one of the AS/400 systems was still running, with only two failed disk drives (in different RAID sets). The other had stopped with a thermal check, and it turned out later to have one failed disk drive. Once we got the A/C fixed and the disk drives replaced, both systems came right up with no issues. We kept using both for another 18 months or so, and had no further problems with them. We definitely prefer our AS/400's well done.

*****************************************************************************************

Oh what a night!

Mario writes:

It was a long dark night. Just the day before, the application owner warned us that we should do a data back up before running the night process. We had not done that since we started using the application, and it had been a year with out any problems. That night we ran the procedure as always with out the pre backup, it happened, the system blew up with a mayor error and we did not have a back up to restore from. We had to enter all the day's data and it took us a full day -- plus what was been posted that day.

*****************************************************************************************

Good idea, or so he thought

Pablo writes:

One Sunday of this year, our operator had instructions to make a *NONSYS backup so that he would type ENDTCP and ENDSBS *all in order to make backup. Before he could do it, he noticed the LAN Console went off, so he had a "good idea." He connected to the i5 from his computer and ran a transfer job to QCTL, and then he typed ENDTCP. He called and asked me why he could not run ENDSBS *all, and why he could not type any command.

*****************************************************************************************

Let me show you what NOT to do

Rob writes:

I was a young systems programmer working for IBM in Kingston, New York back in the late sixties when I received orders from Uncle Sam to report for active duty. I had been asked to transition my 'knowledge base' to the new guy fresh out of college. As a part of that knowledge transfer, John and I were in the computer room one morning where I was teaching him how to accomplish a 'tape dump' on the IBM 1401. I carefully showed him how to thread the tape onto the drive, raise the glass door on the front of the drive, and then hit all the right buttons to get the tape to ready. Having successfully run the program to dump the tape to print, I then instructed him how to rewind the tape for removal. During its high-speed rewind, I sternly warned John to be very careful not to touch the handle on the glass door so as to inadvertently open it. To be sure he understood what part of the door I meant, I very lightly tapped the door's handle. The door flew down - and the tape flew out everywhere. By the time I got things under control, I had an accounting master tape that was stretched, crinkled, folded, stepped on - you get the picture. My jaw dropped to my chest and John's to his knees as he witnessed an object lesson that he, nor I, would never forget. Fortunately, we were able to recreate the accounting master tape that had been destroyed. To do so I had to retrieve the prior week's master tape.

*****************************************************************************************

Blame it on the pink elephant

Larry writes:

Many years back in the System/34 days, I had an operation supervisor who had to pick up her three-year-old daughter from the baby sitter and bring her to work. She had to work extra due to some special processing we were working on, and she was the only person in operations to run the jobs. Everything was going find until the little girl looked up on the face of the System/34 and saw a pink elephant sticker that another operator had put on the machine. Needless to say, the little girl saw it, giggled, and then proceeded to stand on her "tippy" toes and press the pink elephant sticker. Everything went quiet. The System/34 was dead. The operation supervisor's face was ashen white. Under the sticker was the emergency shut down button. The little girl's Grandmother showed up in approximately 30 minutes and the operation staff worked exceptionally late that night. Oh, the "pink elephant" was removed, and a clear cover was put over the emergency shut down button.

*****************************************************************************************

Hold that finger still

Rocco writes:

Back in the 60's I worked at a bank as a computer operator. We had an IBM 360. An electric typewriter connected to the 360 handled the console/system communications. When a typed command was given, you pressed the enter key to acknowledge it. But, the system did not register the command until you removed your finger from the enter key. We were running a posting program that ran for several hours. At about four hours into the posting, I erroneously typed an abort command and pressed enter. As soon as I pressed Enter, I realized the error. I did not remove my finger from the enter key for two more hours.

*****************************************************************************************

Sometimes it's really hard not to laugh!

Pablo writes:

Many years ago I worked in an environment where you see too many bosses and to few workers, one of the bosses ask me... "I have to transfer a file from our mainframe to your A/400, does a megabyte in mainframe system size the same in your system?" I tried many answers, such as "What is it heavier 1kg of steel or 1kg of wood?

*****************************************************************************************

As the saying goes, if it seems to good to be true -- it probably is

Peter writes:

Some years ago, I needed to apply a PTF Cume tape to my Company's F50 machine. The machine was based 40 miles away. From experience, the process would take between six and eight hours. I didn't want to hang around the office on Saturday, so I needed a plan. We'd already rigged up dial up access via a modem and an ASCII workstation controller card for support purposes, so I worked it out that I could load the PTFs during the week, then dial in from home first thing on Saturday to bounce the machine. This would allow me a nice breakfast, time to shop, and then I could stroll into the office, check to see if everything was OK and claim the eight hours overtime -- easy!

So on Friday, I loaded the PTFs from tape, and set them to apply at the next "unattended" IPL. On Saturday morning, I woke up and dialed in to the machine, typed PWRDWNSYS , changed the end option to *IMMED and, not yet being quite with it, forgot to change the restart parameter to *YES - major OOPS. This now meant that I had to travel into the Office, IPL the machine from cold (an attended IPL) and then bounce the machine again to apply the PTFs. Needless to say I actually worked the eight hours overtime and didn't get much shopping done.

*****************************************************************************************

Month end run turns ugly

Reuben writes:

A few days ago, I was performing the month end run, at the same time Group Internal Audits were around to check on programs, fields and files that were recently changed. I had to restore from tape to libraries prelib and postlib. I tried it for the first time, but on setting the parameter to target, I didn't realize that I was restoring files from the pre-changes to the production library. The prelib changes were copied to the production libraries and we lost all the previous week ending data. We had to endure five extra hours, after hours, to rectify the error. And up to date, not all of the data is fully accessible -- and we still have users complaining that they cannot access data from one or two areas.

*****************************************************************************************

The command "Fix" is not all it's cracked up to be

August 2006 Winner

Steven writes:

A few years back, a few programmers and I were sitting around discussing the best course of action to take with a particularly difficult problem. After a rather lengthy banter back and forth, I jokingly commented, "Wouldn't it be nice to just type in "fix" and all this go away." Needless to say, I was also sitting at a terminal at the time and was actually typing that in. I pressed enter and waited for the inevitable "command not found" error, strangely enough, it didn't appear. It seems that somewhere in the original program build for the system; a programmer had created a program called "fix" that re-initialized all of the data files. It was a real problem explaining to management how we had managed to lose about four hours worth of data entry plus half a day reloading from tape. That program has long since been deleted.

*****************************************************************************************

MI causes crash, oh my!

Theo writes:

I started to teach myself machine interface (MI) programming. When everyone went home at 5 p.m., I created an MI program and started the compiler. Within a few seconds, the iSeries crashed. Bad luck I thought, so next day at 5 p.m. I tried to compile it again. Within a few seconds, the iSeries crashed again. The following day, people warned me about the five o'clock syndrome, so I started the compiler during lunch break. Within a few seconds, the iSeries crashed and within two minutes a technical guy rushed into my room asking "why my name was listed a few thousand times in the system dump".

The MI manual states that "If a comment in an MI program is not properly closed, the compiler will crash" and that is an understatement: the iSeries goes down in a few seconds. This can be fixed by adding a line containing all comment-delimiters to the end of the MI source. This was also clearly stated in the manual.

*****************************************************************************************

Slip of the finger brings TCP/IP down

John writes:

Once upon a time, I used the ENDTCPSVR command to bounce the HTTP server. I know you can restart the HTTP server with the STRTCPSVR, but sometimes you need to end it completely before restarting. And of course, the default for the ENDTCPSVR SERVER parm is *ALL. Since I have to type *HTTP over *ALL, the first key to hit is the shift key and then the eight key to get a *. Well, shift is right under enter so it's all too easy to end all TCP servers, including telnet. After bringing down TCP/IP on a customer's machine in the middle of the day, I learned to be extra careful using this command.

*****************************************************************************************

"We are all one keystroke away from humility"

Paula writes:

My first real IT job was with a software vendor who wrote medical applications. I was given a simple project. The project was to dial into over 100 hospital systems and to check the patient file version. Some changes had been done and this was when we had to manually go into the client's system and download the newest version and then rebuild the physical file. The company even had a simple command to do this. But at some point it asks you if you want to clear the file. You always put *no because this is the main file that holds all of the hospitals patient information. Well, I neglected to say *no and ended up clearing the patients file for well over 100 hospitals. They had to use the previous backup to restore the file. My boss told me then -- and I have kept this saying for over 15 years -- "In this field, we are all one keystroke away from humility."

*****************************************************************************************

Never nap during a backup

Bill writes:

A job I had as a college student required me to work third shift and to be the tape jockey for nightly backups. On our new shiny AS/400, I proceeded to start the backups and thought what a good time to take a nap. With the console set at its highest volume for break messages, I proceeded to position myself on a very high pedestal chair with my head against the wall and feet propped up on the system.

Beep... time for a new tape. I proceeded to jump off the Chair, but one problem existed that I was not aware of. My legs were fast asleep – I did a face plant! With my nose bleeding as it had never bled before, I crawled to the phone and called for help -- legs still comfortably asleep.

*****************************************************************************************

Like magic, the AS/400 turns into a System 36

Bob writes:

When I started working with the AS/400, I knew how to work in the System 36 mode only. The people that I worked for never heard of a system 36. The second week on the job, they sent me to an IBM class. When I came back from the class, they told me that their software company told them that I had turned their system into a System 36, as they were getting System 36 console messages that they had never seen before. I was fired on the spot. I consulted a friend who told me that there is a configuration parameter that enables the AS/400 to receive System 36 like messages. All I know is that I am not that smart to reprogram a whole operating system!

*****************************************************************************************

Haughty operator causes self humiliation

David writes:

I had started an AS/400 contract at a company that supported billing systems for two different Telephone companies –- located in three different states. One staff, we had one operator that was very imperious and haughty. She had once worked for IBM and she took every opportunity to remind us all of that fact. She would scoff at anyone's ideas or recommendations as being insipid. So, it was with particular delight that all our IT staff saw this debacle unfold.

The IT manager wanted substantial documentation pertinent to our nightly operations tasks so he'd instructed her to pull up the many commands our shop used, prompt them, fill in appropriate parameters, then hit the print key for each, hit F12 and place the printed sheets in a binder for reference to the new operator(s). The operator was in her cubicle typing away, concentrating on the documentation task. Out of the corner of my eye, I saw her suddenly sit up erectly and start yelling, "Oh my god! OH MY GOD!" I asked her what was wrong. Her words coming out like a machine gun, she replied, "QUICK! HOW DO YOU STOP A PWRDWNSYS *IMMED?" I told her that you can't. She yelled at the top of her lungs,"DON'T TELL ME THAT!" Then I noticed my screen going black. I looked up to see all the other programmers leaning back in their chairs looking at their own blackening screens. Then the phones in the IT manager's office started ringing. Our manager leaned out of his office, and asked us what the hell was going on with the system. The operator stammered, "Ummm… I… I …I was doing the documentation screens you'd requested and I….I …. I was documenting the PWRDWNSYS *IMMED' command and I accidentally pressed the Enter key instead of F12. How do you stop a PWRDWNSYS *IMMED?" The IT manager scoffed, "YOU CAN'T!" The operator was so satisfyingly timid after that glorious day.

*****************************************************************************************

Naughty user uses X option

Jonathan writes:

I was asked by a client to modify a menu option to send a message indicating who was using the option, because we intended to take the option away from all users and wanted to know who was using it. I embedded a call to a CL program that retrieved the user's profile and inserted it into a message sent to the operations manager's profile message queue. This manager wanted the message also to be sent to the operator's message queue, something I thought was unnecessary, but he was insistent. Because I originally thought only he and I would see the message, I foolishly decided to get cute with it, and the message read: "Hello! My name is [profile], and I am a very naughty user: I am using the X option without authorization. Kick my a**!" In hindsight, I probably should have expected that the ONE person still using the option also had reason to look at the operator's message queue, and would see the message, and would be offended by it -- and of course, that's exactly how it developed.

*****************************************************************************************

I was just trying to send an email!

Leslee writes:

When I started my new job I had been out of the loop for about five years, and I had not realized how far the AS/400 had progressed. My first assignment was to create some corporate reports and email them to the users via the AS/400. "Wow, the AS/400 can email!" I thought that was fantastic. I got a book and tried to figure out how to do that. One of the commands was to ENDTCPSRV with a parm for the particular server to end. The default is *ALL. Since I didn't have a clue what TCP/IP was and didn't know which server to end I left the default and hit enter. Unfortunately, I had to issue this command on the production machine and nearly all 7,000 users across the country use TCP/IP to connect to our system. Needless to say, I basically shut the system down for over three hours. I had no idea I had done anything wrong, and I didn't realize I had caused the system to go down; because, after all, I didn't do a PWRDWNSYS. About three hours later, I received a phone call from one of my bosses asking me in a forced calm voice exactly what was the last command I issued and why. It was then that I thought I may have caused the problem. Amazingly I didn't loose my job. Thankfully, their philosophy is you will never get in trouble for trying to do your job.

*****************************************************************************************

If you have the guts, press enter

Rob writes:

Years ago my shift leader was quite unpopular with the programmers. Logical because they always wanted more than we could give, it was never enough. So one day I read a message queue and I see the phrase 'xxxxx is an explicit. xxxxx being the shift leader, of course. Right at that moment that man himself appears and looks over my shoulder. Needless to say that this man -- who was *huge* by the way -- prowled around the programmers department, murder in his eyes. It turned out that one of the programmers had typed that message. Another programmer then said "If you have the guts you send it." And pressed <enter>...

This all was quite innocent, but years later I witnessed someone power-down a full production AS/400 with several thousand users in just the same way. Rob Baartwijk

*****************************************************************************************

Logged off and let down!

July 2006 Winner

Tom writes :

As an AS/400 system administrator, I often borrowed other people's computers instead of going all the way back to my desk to handle problems. One day I signed on using an accounting clerk's computer. I finished my task and figured the user would sign me off and sign back on as herself. When I returned to my desk, the AS/400 suddenly shut itself down. I rushed back to the accounting department and asked the clerk what she had done before the system went down. She smiled and said she found a menu option to log me off, and power down her computer: PWRDWNSYS.

*****************************************************************************************

Good looking operator seeks…. Oh Know!

Doug writes:

I was working a weekend night shift and one of the female users wanted to know how to send a message out to all users. Users had access to the SNDBRKMSG command, so I showed her. She started typing this message about how I was looking for a date and about my physical build, which was exaggerated. Instead of hitting the "backspace" key to remove the message, she hit "enter". Of course, default TOMSGQ is *allws. I had to have security let me into the executive offices where I had to remove their messages from their workstations. As to the rest of the company, they had a laugh. I explained to my boss and fortunately no one got into serious trouble.

*****************************************************************************************

Switch happy electrician brings the iSeries to its knees

Thomas writes:

Not long ago we had an electrician working in our computer room. (Wait let me finish!) It so happens that we do have a UPS. However between the UPS and our iSeries is a breaker panel. The electrician started flipping switches on the panel and (you guessed it) brought the system down.

To add insult to injury, one of our network people went in the computer room to ask the electrician what happened. So the electrician demonstrated by flipping switches again! This brought down some more servers.

*****************************************************************************************

"Cool" boss ends up in the "hot" seat

Andrew writes:

I started 30 years ago as a trainee programmer on a S/3 (model 10 to be precise), I've been on them all since, S32, S34, S38, S36 AS400 et al. Our computer room, complete with an MFCU1, had big red mushroom emergency power cut off buttons on each of the four walls. Now our IS manager used to like to show visitors around our towering machine and explain the card unit, the removable disks. He also liked to look cool. So yes, one day looking cool he just leaned back.

*****************************************************************************************

TRM means terminate – not trim

Doug writes:

This "oops" occurred with one of our midnight shift operators. He thought he would do us a favor by speeding up our end-of-day batch processing program. He did a "trmsbs" with an *immed option. He realized something wasn't right by the abnormal end break message, so I was called. I asked why he did the command and he said he thought it meant "trim subsystem" and I guess he never looked at the screen carefully and would have seen TRM means terminate.

*****************************************************************************************

24/7 distribution center accidentally shut down!

Steven writes:

Several years ago, I was trying to delete a message description that someone had placed in the QCPFMSG message file. I must have hit the Enter key because when I put in the four to delete, I deleted the enter message file. The AS/400 locked up, and we had to call IBM to restore the operating system. Because I failed to check the delete confirmation screen, I shut down a 24/7 distribution center for hours.

*****************************************************************************************

Operator prints first and asks questions later

Mike writes:

Years ago we upgraded the operating system of our system 38 -- over the weekend. The following Monday night, our night operator was told he could go home after he printed everything in QPRINT. The night operator wasn't aware of the save spooled file feature offered in the new release and repeatedly released all the reports in the queue. After hours of printing, the operator became suspicious that something wasn't right and went home. He ended up printing over 10 sets of the reports which consumed several cartons of green-bar. Needless to say, the following day the night operator reviewed the list of enhancements from the new release.

*****************************************************************************************

Temporary clerk could equal permanent problem

Joe writes:

While working for a major clothing manufacturer, we had a shortage of staff and some clerks were brought in at night to help sort and decollate reports. First rule for them: Touch nothing but paper!

The operators used to joke that on CL messages 'C' in (C D I R) meant continue. When a job broke with a "could not allocate object" message, the temporary clerk typed in 'C' and blew away the nightly billing.

*****************************************************************************************

"C" (C D I R) strikes again!

Ira writes:

I had JUST gotten on board with the AS/400 and had taken a C10 (around 1991) to a trade show to demo some software that a contractor had written for us. We had a problem with the box and had to load some licensed programs at the last minute -- from slow QIC TAPES. I got the message (C D I R) and said to the contractor, "C to continue," and he replied, "C means Cancel!" and I repeated "C to continue," thinking that he MUST be kidding, and he replied, "C means Cancel!" and, convinced he WAS kidding, I hit "C" and pressed Enter. The whole three hour load had to be done again and we were VERY late with setup... I will NEVER FORGET - "C" = CANCEL! Since then I have never made any other mistakes.

*****************************************************************************************

Personal SNDBRKMSG message leaves programmer red-faced!

George writes:

Over 20 years ago, I was the senior programmer in a System 38 shop. One of the other programmers and I used to like to communicate via SNDBRKMSG. One day, as one of the very attractive female staff walked past I started to send a message to the other programmer commenting specifically on how good two of her attributes were. As my finger was on the way down to press the enter key I noticed that I hadn't changed the default from *ALLWS (which was the default back then) just as I pressed the enter key. Everything slowed down then as I started to consider the humiliation I was going to suffer as my message, with my name on it, was sent to every terminal in the building.

I did the first thing that anyone in my position might have done, I turned the power off to my terminal. I got up from my desk and instructed all the programmers to go to different sections of the building and to clear all the message queues as fast as they could and with that they all disappeared. Normally this action would have earned me a "Why" question but I guess the look on my face and the sound of my voice told the story. Seeing my life dissolving around me, I ran into the computer room and cranked the System 38 console switches around to "enable" and "CPU stop" and pressed the load button. I always wondered what would happen if you did that. This gave me some thinking time and a chance to make sure that all my programming buddies were in position.

Bracing myself for the onslaught I pressed the load button again on "CPU Start" and hoped the guys would clear the message queues quickly. After about a minute, I get a call from one of the guys telling me that there are no messages to clear and a similar message from another so I call them all back. I check the joblog for my terminal and find that turning the power off before the command completed had saved my bacon.

"What was that all about?" quizzed my colleagues. "Just seeing how well you can handle yourselves in an emergency" I lied. Meanwhile the operator was going berserk over her perceived violation of her computer. She told the boss what I had done when he got back from lunch. "What's this all about?" he asked me. "It's a long story" I murmured as he invited me into his office.

The lesson I learned that day: never send any communication that you didn't want pinned to the notice board!

*****************************************************************************************

Know-it-all really doesn't know that much after all

Parker writes:

There are several IBM commands that give you more options after you press the Enter key. One of my coworkers who "knew more than I did" was showing me (on a production machine) the additional parameters after you hit the Enter key on the PWRDWNSYS command. As I was trying to tell him that there weren't any additional parameters, our machine was on its way down. There is no way to stop PWRDWNSYS once you hit enter. He had a pretty red face explaining that to upper management.

*****************************************************************************************

Is your office klutz proof?

Vickie writes:

We had a problem with our LTO tape drive and IBM sent out our service representative with a new one. While he was in the computer room installing it, our system went down (this was the middle of the day with everyone on the system). I was notified by several of our users and went in to see what had happened. After a couple of minutes of questioning, the service representative finally admitted he had knocked out the plug from the back of the iSeries. It turned out he had almost dropped the tape drive and as he grabbed for it to prevent it from crashing to the floor, he knocked out the power cord on the iSeries. What was amazing to me was that he didn't try to contact me about the situation. It was as if he thought that since he plugged it back in right away nobody would notice. To add insult to injury, he then tried to insinuate it was our fault because the plug wasn't "covered". I guess we should have been prepared for a service representative dropping a tape drive behind it --the back of our iSeries is against the wall.

*****************************************************************************************

Hey, how did that CD-ROM drive get there?

An anonymous user writes:

It was my first time loading a Cumulative Fix package to an iSeries, which, in my own defense, was new to our shop. I carefully read all of the instructions and PTF cover letters and went into the computer room to load the PTFs, but each time I checked after executing the command, no PTFs had been loaded. After doing this multiple times I finally called IBM tech support. The support person had me go through the same steps without any success. Finally when asking a question about activity lights on the CD drive he asked what was on the panel display. When I told him I had to put the phone down to go look he realized that I had the CD in the drive on our PC console. I didn't realize there was a CD-ROM drive on the 400 itself.

*****************************************************************************************

Creative programmer leaves his finger "print" behind

Judy writes:

This creative programmer designed a program for the menu option screen. When a menu option was entered, the screen would slowly start raising an image of the programmer's middle finger. As they say, it's all fun and games until the boss finds out. Years ago we had a programmer who would come up with creative programs on our menu option screens. When a menu option was entered, the screen would slowly start raising an image of the programmer's middle finger -- from the bottom of the screen to full screen. This menu option was not used but once a year so by the following year the programmer had left our company and forgot to remove this program from the system. Well one of our company directors happened to be running the menu options for the year and imagine his surprise at what appeared on his screen. Needless to say we had a lot of explaining to do once we found out who created the program.

*****************************************************************************************

Where was that Save file created?

Richard writes:

A while ago an IT manager at one of our customer's shops wanted to delete a library. He first created a save file and performed a SAVLIB to the save file so he could restore the library back if required. Then he deleted the library without a problem. Unfortunately, he created the save file in the same library that he was deleting, so he lost the lot!

*****************************************************************************************

Hot food turns iSeries cold

Daniel writes:

Our break room is located just outside our server room. One day at lunch I tried to use the microwave, but it seemed that the breaker was flipped. I couldn't find the breaker and was hungry, so I decided to move the microwave to the other wall and try it there. As my food heated, I popped into the server room for a second. Something seemed eerily wrong -– it was quiet. No fans, no lights, nothing! Although our servers were on a UPS, I had managed to plug into an outlet on the server side of the UPS and flipped a breaker there. I ran to the breaker box for the UPS, but unfortunately I can't travel at the speed of light so I spent the rest of the day sheepishly explaining why we had an hour outage that day. We've since disabled the rogue outlet.

*****************************************************************************************

Statically-charged admin creates hairy situation

June 2005 Winner

Timothy writes :

Early in my career, I was working as a systems administrator/programmer for a company that had a small B model AS/400. Those units came with a 1/4 inch tape drive in the front. Well, my problem begins, strangely enough, with fashion (or lack thereof). As a low-paid administrator, most of my pants were polyester Wal-Mart specials. The problem is that my legs are hairy. So every time that I walked, I generated static electricity...even on the linoleum floor of our IT department. One day, while taking the tape out of the drive, a static spark jumped the 1/2 inch from the metal frame of the AS/400 to my finger. Not only did it numb my arm for the next 15 minutes, but is fried the UPS in the AS/400, crashing it immediately. It took 24 hours for IBM to get a part in and installed. Needless to say, though my boss understood, he got an anti-static mat for in front of the AS/400 and required me to stand on it when I changed tapes.

*****************************************************************************************

Change to Key Fields results in an all-female jury

Mary writes:

Just out of college, I began my first computer related job at a County Computer Center. It had both a Sys/38 and AS/400 with various systems for the County departments: Jail, Courts, Jury, and Voter Registrar. I completed my training and was assigned the task of building a new jury pool. I was simply reviewing the system when I noticed that a LF didn't match the actual File Object Key fields. Thinking I would just correct the issue, I recompiled the LF from the current DDS. I neglected to look at the usage of the LF and evaluate if the new Key fields were actually appropriate. It turns out that the LF in question was used in the weekly juror selection and the change to the Key list resulted in SEX being the first Key Field for the process. Unfortunately, when the Juror notices went out, no one bothered to look at the reports that clearly contained all "F"emales. Needless to say, they didn't have court that week and I was lucky to keep my job.

*****************************************************************************************

May 2005 Winner

How many programmers does it take to speed up testing?

Terry writes :

Many years ago there were a number of programmers gathered around the console of our development System/38, trying to do determine how to make the testing run a little faster. One of our programmers had the great idea to have fewer active jobs run. The idea being that the processing would speed up. So, having all agreed that it was a sound idea, we changed the system value to have *MAXACTLVL of 1. Needless to say, we realized or mistake almost immediately, but were forced to wait for several hours until the console freed up so that we could change it back.

Luckily, as I said, it was on the development system, not production. *****************************************************************************************

April 2005 Winner

Where has all the data gone?

Michael writes :

An operations manager at one of our software development clients was confused by a couple parameters on the nightly SAVOBJ command and specified storage (*free) instead of clear (*all). The next morning everyone in the place wondered where all the data had gone.

*****************************************************************************************

Even the electrical engineer missed the switch!

An anonymous user writes:

At a previous company, we arrived on Monday morning to find the QSYSOPR message queue full of messages about "Mirror suspension this" and "Mirror suspension that." A quick call to IBM had an engineer on site right away. He walked into the data center, straight to the iSeries and switched on the expansion units.

We had switched off all the machines in the data center as a diesel generator was added to the UPS. My boss -- who had been an electrical engineer -- switched them back on again and didn't realize that the big white button didn't power on the expansion units. Then again, I had missed it, too. The disks re-synched themselves and off we went, not missing a beat.

*****************************************************************************************

A hard shut down and the iSeries is a recipe for disaster

Jerry writes:

We had a 3000 watt Unity UPS system that our iSeries was plugged into. It had a serious flaw that I discovered one day. On the back of the tower unit is a switch that will kill the power to all the outlets on the unit. It was not a rocker switch, but a toggle switch that was very easy to flip. One of our networking people was working behind the unit on some of the wiring and I leaned over the unit. I happened to bump this toggle switch, instantly bringing down our iSeries. Needless to say, a hard shut down is not the best way to bring down an iSeries. It took over six hours to get everything rebuilt and to a point where we could function normally. That same day we had a steel cover built and installed over that toggle switch.

*****************************************************************************************

March 2005 Winner

The flashlight is to blame!

Jim writes :

Years ago, back in our white box days, we were going through one of many system upgrades. Our backups took about six hours at the time. We had shut the machine down and started the backup process. Our customer engineer came in about five hours into the backup. Trying to make good use of the time, he was checking on some cables and card locations. He always carried a "Mini-Mag" flashlight --aluminum, but fairly hefty, none-the-less. Apparently the pocket protector was extra slippery that day; as he bent over to check another location, the light slipped out of his pocket and scored a direct hit on the emergency breaker. It got VERY quiet -- partly from the sudden lack of machine noise and partly from the silence of the customer engineer! So, not only did we get to start the backup over from scratch, we got to go through a long recovery from the abnormal shutdown. Needless to say, we were late getting home for supper that day.

*****************************************************************************************

Changing system values

Robert writes:

We were working on increasing security on our iSeries and in order to increase supported password lengths and support numbers in the first position, I changed the system value QPWDLVL from 1 to 3. About a month went by before we IPL'ed the machine and the new value went into effect.

It seems our custom sign-on screen -- that my boss was very proud of and every employee was used to seeing in the morning -- had a password field that supported ten characters only. When the system IPL'ed, it didn't like the custom sign-on screen because it wanted a password field that supported 128 characters, so it defaulted to the IBM supplied sign-on screen.

We spent about two hours and two IPLs trying to figure out why the system 'all of a sudden' didn't like our custom sign-on screen before it hit me that maybe changing the QPWDLVL value last month might have something to do with it. My boss was very quiet for the rest of the day. I'm much more cautious about changing system values now.

*****************************************************************************************

February 2005 Winner

You might want to think twice before hitting enter

An anonymous user writes:

Virtually everybody in the local iSeries IT shop has done this at least once during their careers. Use the prompter on the PWRDWNSYS command and inadvertently hit enter.

But we have one particularly creative person who TELNET from one system to another, then TELNET to another, then TELNET to yet another. This enterprising soul needed to do a PWRDWNSYS on one of our test systems. Unfortunately, he lost track of what system he was on and did PWRDWNSYS on three production systems before he realized he wasn't on the test system!


This was first published in January 2007

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchEnterpriseLinux

SearchDataCenter

Close