I sometimes use an analogy to a bag of sand. I use it to refer to treating something in the aggregate. Each grain of sand can be inspected for mineral content, density, porosity, size, color, etc. but usually we just talk about how much of it there is. In software people sometimes use a similar technique to deal with customers, resources (oh, I mean people 
), schedules, etc. Treating things in the aggregate is sometimes necessary but never forget what you are losing in doing so.
Today I am excited. Maybe sometimes I’m excited by small things, but, none-the-less, I am excited. Some months ago, I wrote a post on measuring the quality of a service where I tried to articulate a customer focused view of quality. For the last couple of years I’ve had a “north star” of building a great way of measuring the quality of the experience our customers have on a customer by customer basis – rather than treating the customers as a “bag of sand” and only looking at the health of the overall service. We’ve been gradually making progress on that journey and today is a milestone.
Now for a brief aside before I continue…
9 months or so ago, we created something we call the “Top Customer” program for Visual Studio Online. We measure activity in the service and identify the top N customers as defined by # of active users, depth of usage, etc. We then offer this set of customers participation in the Top Customer program – some accept and some don’t respond. It’s all good. The benefits of the program for the customer include a direct contact in the product group to help with any issues they experience, early access to new features (think kind of like traditional Beta programs), etc. The benefit for us is that we get feedback on important decisions and early features before we roll them out broadly.
Now, back to the story on quality of service…
We recently (like within the last week) actually produced a quality of service report that actually lists quality of service by account. And then in the last couple of days, we composed that with our Top Customer program to see if any of them were experiencing issues. Sure enough it seemed so – about 4 out the the top 30 or so seemed to be getting experiences of lower quality than we expect to be able to deliver. So we contacted them to ask if they were seeing issues themselves and today, we got the first response back. The response we got back included this:
“I had put this down to general connectivity/service issues, but if you’re saying that you’re seeing this your side then yes something must be amiss. The reports I’ve had from devs are slow pull requests / more than normal commit fails and general authentication issues. Nothing so severe that we’re unable to operate, but is requiring an extra retry or request.”
As my 16 year old daughter would say BAM! We identified an issue a customer was having before they were frustrated enough to contact us to complain about it. I’m incredibly happy about that. I suspect we’ll need to continue doing this kind of cross checking for a little while but, ultimately, we don’t want to even bother the customer with it, of course – we just want to fix it for them proactively.
I also ultimately want to evolve our service health dashboard on http://www.visualstudio.com/support/support-overview-vs to provide a personalized view so you can see how your individual account is doing rather than the entirety of the service.
It’s a journey and we’re not nearly done yet but I’m really excited about this milestone.
Brian