Thursday, December 31, 2009

Saying Goodbye To 2009

I'll close out 2009 with an update on the goals I set at the beginning of this year and things that I've accomplished. Call it a bit of shameless self promotion, but this is my blog after all. :-)

  1. Learn…
    • Powershell. Although I dropped using the command prompt in favor of a Powershell prompt, I really haven't done anything significant with Powershell this year. I fell short on this one for sure.
    • More about DMVs. I've written several DMV queries that have become part of my toolbelt and gave a presentation about DMVs at SQL Saturday Orlando in October. I met this goal, but there's plenty more for me to learn.
    • SQL 2008. I've deployed a handful of SQL 2008 instances to my company's production environment and picked up plenty of tidbits here and there from blogs, articles, etc.. I'd say I met this goal.
    • Analysis Services. I took an online class with Brian Knight in the fall and have dabbled a little bit with SSAS, but I wouldn't say I've done anything significant enough to consider this goal met.
  2. Be a better mentor. Gave presentations at OPASS and Space Coast SQL User Group, Orlando Code Camp, SQL Saturday Atlanta, Jacksonville, Orlando, and South Florida, and presented at the 2009 PASS Summit. Also published an article on Looking closer to home at my coworkers, I'd like to think that those around me learned from me (though they're better suited to say if that's the case or not). I'll call this goal met.
  3. Publish twice a week. I published 64 blog posts this year (65 including this one). That's a tad short of the 104 I was shooting for. I guess you could say I went for quality over quantity. I'll call this one halfway met.
  4. Go to the 2009 PASS Summit. Check! I went to the Summit this year as a speaker and had a fantastic experience.
  5. Run a half-marathon. I failed miserably on this one. I ran a 5K in the spring but stopped running over the summer. However, my 8 year old son has expressed an interest in running so we've started running together. Even if I fell short on this one I still come out ahead because any exercise I've done is better than no exercise.

Final count – 4.5 out of 8. Not bad but always room for improvement.

I hope that you had a great 2009 and look forward to an even better year in 2010!

My Biggest Weakness

I took some time off in December and I'm finally getting around to catching up on RSS feeds. Turns out I’ve been tagged by Ken Simmons to contribute to another chain post to answer the classic interview question "What is your biggest weakness?" I've been told in the past that the best way to answer this question is with something clever – like a strength that you call a weakness but you really spin into some form of self promotion to make yourself look better. But…I don't think you really want me wax poetic about how great I am (I don't want to either!) and this isn't a job interview, so I'm going to be honest because the only way we really learn how to be better is to recognize our faults and learn from them.

My biggest weakness: Letting go. Let me explain…

  • Letting go of opportunities – in other words, knowing when to say no. It's especially tempting to say yes when people ask you to do something that sounds exciting, but taking on too much can be detrimental. I think it's far better to be very successful at a few things than to take on too much and fail.
  • Letting go of responsibilities – in order to do new and exciting things you have to learn how to hand off responsibilities to other people. Think about it this way – at some point someone had to hand off their responsibilities for you to be in the position you're in today. This goes back to what I just said about taking on too much – if you just keep adding things on you're bound to fail at some point.
  • Letting go of being right – compromise can be difficult to accept and we all like to be right. The reality is that we're not always right and it's better for our personal and professional development to learn when to hold your ground and when to admit that someone has a better answer than you.
  • Letting go of things I cannot control – my friend Andy Warren recently told me "you can only control 50% of things" and it really stuck with me. There's no sense in spending time worrying about things you have no control over. The more time you spend on the things you can't control, the less time you have to focus on the things you can.

I've learned to deal with this particular weakness partly by learning from those who have "been there and done that" before me, and partly through experience (a.k.a. the school of hard knocks). It's not always easy letting go of things, but looking back at where I've been and where I am now I realize that it's something I've got a much better handle on now.

Thursday, December 3, 2009

Fix The Problem, Don't Just Band-Aid It

Let's face it, when it comes to computers there's a 100% certainty that something is going to break eventually. Maybe it's going to be a hardware problem like a drive failure, maybe an OS patch will break something, or maybe a new release of software contains a memory leak. It's this kind of stuff that's part of the reason production guys like me have a job; After all, if everything worked perfectly all the time there would be no need for a production DBA, right?

Throughout many years of working in production environments I've seen my fair share of problems. I've also seen many different ways that people chose to fix them – most good but some very bad. Perhaps the biggest mistake that I see repeated constantly is when people "fix" a problem with a band-aid or some kind of workaround. I submit the following examples:

Problem: A memory leak in a custom ISAPI DLL causes IIS to stop responding to requests after 8-10 days.
Solution: Dev can't reproduce the problem. Reboot the affected servers once a week. Dev is working on a complete rewrite of the software that will be out in 3 months anyways and this will go away.

Problem: An application log file is filling up all available space on the drive it lives on.
Solution: Move the log file to a drive with more space.

Problem: A SQL Agent job is failing intermittently with an out of memory error.
Solution: Set the job to retry 10 times with a 5 minute wait in between retries.

The issue I have with each of the proposed "solutions" to these problems is that they are not solutions at all – they're band-aids. In real life when we get a cut or scratch and put a band-aid on it all we're doing it masking the problem while our bodies work on the real fix underneath. Similarly, when something on a server is failing and we employ a workaround all we're doing is masking the real problem.

In the first example, the problem was actually happening on 80 servers. Asking the production team to reboot each once a week meant that one person was spending an entire day out of each work week to do the reboot. Further, since dev couldn't reproduce the problem that means they didn't understand it; how then could they guarantee that their rewrite would fix it? And what if the project timeline slipped from 3 months to 6 months because of feature creep, bugs, or mismanagement?

In the second example, the application log was growing because of database corruption. Moving the log file to another drive would have just given it more room to grow until the corruption was fixed.

In the third example, the memory errors were due to the fact that no page file had been configured in the OS. Setting the job to retry may have allowed it to eventually run successfully, but the real problem would have just manifested itself in other (potentially worse) ways.

To be fair, I'm not against using a workaround…as long as you continue working on a real solution to a problem. What I am saying here is don't consider a workaround a solution. When things break it's usually for a reason so don't ignore what your servers are trying to tell you by masking the problem and calling it fixed.

Wednesday, December 2, 2009

What It Means To Be A Production DBA

The Bobs Remember that scene from Office Space where the Bobs ask, "What would you say you do here?" I've been a production DBA for most of my 10+ year career and I've boiled down the non-technical answer to "I keep the lights on and the wheels turning". While it may be an adequate answer for a casual conversation, it's far from a fair description of what I do on a daily basis.

Recently I came across a blog post by Tim & Lori Edwards (Twitter: @timbo_b_edwards and @loriedwards) that I thought does a great job of answering that question and is worth sharing: What Is A Production DBA, Anyway?

Tim & Lori's blog – SQLServerTimes2 – is relatively new but they've got some good posts up so far. Go check out their post on being a production DBA, and while you're there take a minute to read their other posts and consider adding them to your RSS feeds.