Stop, think, research, debug

I got this great letter from a listener/reader recently. They listened to a recent show on Debugging Windows and it helped them debug a problem at work, but not in a specific technical way. Instead, it changed how they thought about their approach to the topic.

By the way, I've had some amazing guests on lately. If you haven't subscribed, it's a good time to join us. Explore the archives and check our our diverse topics and voices and subscribe. Also, BTW there is a new This Developer's Life out so check that out also.

I've been doing Hanselminutes: Fresh Air for Developers for almost 500 episodes over darn-near 10 years. Getting emails like this is so meaningful, especially when I think about taking breaks or stopping. Sometimes a few shows will go by with no comments and I'll wonder if anyone listens, and then I hear from a Real Live Human who was helped by all this free content I put out and it keeps me going. So first, thanks for this, if you've ever emailed or donated to fight diabetes.

Here's what they said, with emphasis mine.

Scott,
After listening to your podcast with Mario Hewardt earlier this week on Windows Debugging, I had some of the things you were talking about running through my head. Though I've always come away from your podcasts feeling enriched and excited to tackle new and interesting problems, this was the first time that it had a direct impact on my work so soon after listening.

I work at a big data company that does a lot of social network analysis. We use ElasticSearch in our stack, and we are consistently processing millions of documents using complicated, user generated queries. A release we put out late last week allowed for many, larger, even more complicated user queries, which in turn led to substantial slowdown of our product. Though the code only existed in our staging environment, we are on a deadline for release early this next week. As it became obvious that the application was spending a LOT of time in the code my team was responsible for, we were tasked with "fixing" it ASAP.

I took the first shift, and though my brain immediately started coming up with ways to improve our code, something about your podcast regarding "know the tools your tools are built on" was stuck in my head. Instead of jumping in and optimizing what I was already comfortable with, I spent an hour researching the internals of the ElasticSearch functionality we were relying on.

Not sure how familiar you are with ES, but it distinguishes between searches that simply return a set of documents that match a query, much the way that traditional SQL databases do, and searches that return how well documents match a query, for ranking purposes. As it turned out, we were inadvertently using one of the latter ones, meaning when we provided X giant queries in an OR block, even though it was an OR block, which we expected would short circuit as soon as it returned a TRUE condition, it processed all X queries to determine how well each document matched. My big O notation is a bit rusty, but suffice it to say, it was one of the bad ones.

Instead of a gigantic fire drill app optimization over a weekend, it turned out to be an hour of research followed by switching the word "bool" to the word "or". It's remarkable how the most efficient coding you can do is often stopping and thinking about the problem for awhile!

Anyway, thanks to both you and Mario for saving me and my team a bunch of time!

This was a great reminder to me as well. Research is hard. It's not as dynamic as interactive debugging but it can often save you many wasted hours. Truly successful debugging means doing whatever it takes to understand the problem domain and the code paths.

Do you have any tales of debugging where taking the time to really understand the problem domain saved you time? Or perhaps the opposite, where you just dove in and poked at some code until it worked? Don't be ashamed, I think we've all be on both sides.

Sound off in the comments!

Sponsor: Big thanks to Infragistics for sponsoring the feed this week. Responsive web design on any browser, any platform and any device with Infragistics jQuery/HTML5 Controls.  Get super-charged performance with the world’s fastest HTML5 Grid - Download for free now!

* Photo - US Navy Cryptanalytic Bombe by brewbooks, used under CC