Big Data: A means to and end, or an end in itself?

Big Data: The Matrix in Reality?

Back in the spring of 2011 McKinsey declared “Big Data: The next frontier for innovation, competition and productivity. As usual, they called this explosive trend “Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers.”

Their estimations of value creation were nearly unbelievable back then. They declared big data a critical factor of production, along with labor and capital. They cited five broad ways using data might create value, from making information transparent and usable at higher frequency to improving performance and making better management decisions.

Experts cite the fact that many firms have jumped on the big data bandwagon -analyzing large streams of information - only to fall into common traps and have nothing to show for their efforts in the end. Some 44% of IT professional surveyed by business software firm Infochimps Inc. said they had worked on big data projects that ended up on the scrap-heap.

Last year the Wall Street Journal reported on “Big Data, Big Blunders” and cited five common mistakes firms make and ways to avoid them. # 1 was “Data for Data’s sake.” Many firms fail to agree on the most important question: What is our goal for use of this information?” Considering anything with an on / off switch throws off “data exhaust” there is a nearly inexhaustible source of data – but to what end?

A lack of qualified data professionals is also causing problems. The Harvard Business Review called “Data Science the sexiest job of the 21^st Century” back in 2012. They say that demand has raced ahead of supply in the data scientist labor pool. The problem is magnified by the lack of formal training / degrees in data science, little consensus on where the role best sits in an organization nor how data scientists can add “the most value” possible. McKinsey estimated a short-fall in the US of 140-190,000 skilled analytic professionals, along with 1.5 million managers who can make better decisions based on their findings.

Marketing and IT departments discussing Big Data.

Another pitfall the WSJ identified as “organizational infighting” or “territorial spats” between departments over who owns a project. I have seen first hand knock down, drag-out fights in Fortune 500 firms between IT (who claimed ownership of the data) and marketing (who desperately needed the data to inform and measure business plans). 

Some claim a Chief Analytics Officer position is the solutions, but that person will be little more than a referee at an unruly sports match unless they control the budget allocation for data analytics.

One of the best uses of big data is IBM Watson – a big experiment in cognitive computing that searches massive amounts of data and serves up hypotheses (with confidence levels) on questions it is asked. And it is designed  to “learn” as it works with more data on a give subject matter.

In the end I am in the camp that supports use of big data as a means to an end.

For business - use of big data and analytics should help save money by defining the most cost- efficient ways to do things; drive incremental revenue by helping identity new business opportunities or optimization of investments by better measuring ROI of all sort of things from marketing spend to distribution strategies. 

Using big data as a means to an end supports one of the most pragmatic business lessons I ever learned: Test – Learn – Apply. Try different approaches, measure their success (or not) using data, and apply analytic-based learning to inform future efforts. Rinse. Repeat. Look smart and drive profitable growth in the process.