This morning’s Observer column.
As the “big data” bandwagon gathers steam, it’s appropriate to ask where it currently sits on the hype cycle. The answer depends on which domain of application we’re talking about. If it’s the application of large-scale data analytics for commercial purposes, then many of the big corporations, especially the internet giants, are already into phase four. The same holds if the domain consists of the data-intensive sciences such as genomics, astrophysics and particle physics: the torrents of data being generated in these fields lie far beyond the processing capabilities of mere humans.
But the big data evangelists have wider horizons than science and business: they see the technology as a tool for increasing our understanding of society and human behaviour and for improving public policy-making. After all, if your shtick is “evidence-based policy-making”, then the more evidence you have, the better. And since big data can provide tons of evidence, what’s not to like?
So where on the hype cycle do societal applications of big data technology currently sit? The answer is phase one, the rapid ascent to the peak of inflated expectations, that period when people believe every positive rumour they hear and are deaf to sceptics and critics…