Monday, June 02, 2008

Lessons Learned

Lessons learned are like
Bridges burned
You only need to cross them but once
Is the knowledge gained
Worth the price of the pain?
Are the spoils worth the cost of the hunt?
- From Dan Fogelberg’s “Lessons Learned”

I've been in the business arena for a long time now; long enough to reinvent myself and change industries several times. In doing so, I've built up a habit of occasionally taking the opportunity to pause and reflect on some of the things I’ve learned through experience. I spent most of the past few days doing just that.

Most of my learning over my career has come to me through making…shall we say…less than optimal choices. In other words, I’ve learned a lot over the years as a result of dumb choices and boneheaded moves. These lessons from the school of hard knocks continue to come to me today (does that make me a life-long learner or just a little dense?) in areas far beyond the scope of this blog. But those lessons are the ones that stick with you, so I wouldn’t trade the experiences if I could.

I’ve personally found the same concept to be true with Oracle Apps: the best lessons are those learned as a result of royally messing things up. I’ve always maintained that you haven’t really “earned your stripes” with Oracle until you’ve put your job on the line at least once by making a huge big mistake or two with the apps (you know, mess up the year-end financial statements, bring down the production instance for a few days…something really big).

Now, you don’t need to make the same mistakes I made and suffer the same pain I suffered in order to learn the lessons I learned. Many of you out there are smart enough to learn from the mistakes of others. In fact, I think the Oracle user community would benefit greatly if we shared our failures and mistakes in the same way we share our successes (although I doubt that our foul-ups will ever make the PR Newswire). While many of us are too embarrassed to share our failures, think how much better off the overall user experience would be if we could prevent others from making the same mistakes we’ve made. I believe this so strongly that I’m even willing to go first (at least first as far as this particular discussion is concerned). So, let me hang a few painful Oracle lessons learned over the years out there right now. I’ll just share the gist of the lessons, but leave the accompanying background and questionable decisions on my part to your imagination (please keep the laughter down - people around you may be trying to work):

  • In the E-Business Suite Financials, unresolved cost transaction errors are essentially unrecorded inventory. The minute you resolve those errors, the costs will hit the books…in the period in which the errors were resolved. Delaying resolution to future accounting periods is a great way to tank the company’s stock or get yourself a new CFO.
  • Writing records directly to apps database tables with custom code is never a good idea, as those foreign keys can be really tricky!
  • You may want to consider testing patches before you apply them to your production instance.
  • Did you know that Oracle writes manuals on basic concepts as part of their product documentation?
  • Never go live on a product with a version number ending in “0” (yeah, I know everybody knows this…or, at least, everybody knows it now).
  • The first person to respond to your SR is rarely the person with the answer to your problem.
  • A name is not a good candidate for a primary key.
  • The biggest issue with custom application performance is a lack of general understanding about bind variables.
  • Exiting a form and killing a query are two separate things.
  • That error message rarely means what you first think it means. Don’t jump to conclusions.
  • Telling a support analyst that you "...could throw a dart on a bus and will hit somebody smarter" will not get you SEV 1 status.

So, there you have it…I’ve hung some of my dirty laundry out in the hopes that someone else will learn from it and avoid the mistakes I made. How about you? Any pithy lessons you’d be willing to share?


Brian Bent said...

I once brought a production system down not once, not twice, but three times in one day trying to make my .profile smart enough to tell me what cluster I was on. I should have put two and two together the first time after I couldn't login to the OS under my ID and production went down shortly after. Running an OS out of open files not such a good thing, especially for the unlucky customers trying to return rental cars that day.

David Haimes said...

I've spent over a decade in Apps development at Oracle and my mistakes are also known as bugs ;)

justadba said...

More than once, my UNIX SAs have done an excellent job of testing the backups due to mysterious "network" errors that kept the database unavailable until the restore was complete. Good SAs like that are hard to find anymore...:-)