In Praise of Small Data
We've been working on a research project for a small client this week and it's been a very pleasurable experience. There’s nothing like asking real people real questions when you want to get real, actionable intelligence about a business problem.
Between phone calls we've been musing on the subject of research and analytics generally, most especially the hype that’s surrounded “big data” the last year or two. Truth be told big data is somewhat over as technology trends go (the latest being the “internet of things” but that’s another post). It seems like every week there's an article about some large organization or industry group expressing disappointment with the promise of big data.
That shouldn’t come as too much of a surprise. There’s no question that advances in technology have made it possible to capture and at least to some extent crunch the vast amounts of consumer data flowing through our phones and tablets as we move around, make purchases and use social media. So there’s no problem compiling big data. The trouble is using it.
Because a mountain of raw data doesn’t volunteer opinions. The great illusion about big data for many business people was that once the data was captured the servers would start offering up tactical recommendations on their own. “Hey Bob, tweet now about rolled oats and sales will be up 14% by Thursday.”
The reality of course is that big data, just like any other data, has to be interpreted before it can be used. Drawing useful conclusions from mass amounts of raw information takes effort, probably a few full-time statisticians and a good deal of time and money. And of course there’s no guaranteeing that once all the analysis is done (even assuming it's good analysis) the result is going to be of any practical use.
“This sine wave shows our customer retention rate flux over the most recent 20 month period.”
Er…is that the question or the answer? Obviously we don’t mean to imply that big data is useless. In narrowly defined circumstances, for example scientific research, it’s proven itself to be almost boundlessly useful. However when it comes to delivering the marketer’s fantasy of a predictive mathematical model of consumer behavior, it’s a flop. The days when we’ll be able to truly define the laws of "social physics" — when we'll confidently deploy message A to influencer B through device C and get transaction D — are a long, long way off. In fact we’ll probably never get there.
Which is why we're so pleased with our (very revealing and useful) small data. Because business, at rock bottom, is all about human relationships. Ask human beings questions about your product and they’ll respond — and in the bargain probably offer up answers to some questions you didn't even know you had. Which is something you'll never get from a server farm.