Is Hadoop dead and should you care?

Google has claimed it now rarely uses Hadoop, the open-source standard for big data, and has launched new technology that creates consumer insights in real time. Should marketers expect the dawn of a new big data era or just of a new Google sales pitch?

Michael Barnett

In case you’ve never heard of Hadoop (or if you have but never quite worked out what it is), in simple terms it’s the framework used to process multiple data sets in batches across numerous servers. It originated from Google’s own MapReduce data tool, which still powers it, but has since grown into a much bigger open-source project under the non-profit Apache Software Foundation, with various vendors selling their own versions.

But now Google is implying that Hadoop has passed its use-by date. At the company’s I/O conference last week, senior vice-president of technical infrastructure Urs Hölzle said: “We don’t really use MapReduce anymore”.

He used the keynote speech to introduce Google Cloud Dataflow, which, it is claimed, is better at analysing data on the fly rather than in batches, producing real-time rather than retrospective insights. To illustrate the practical implications a demonstration showed how, by processing around 400 new tweets each second (which is apparently more impressive than it sounds), Cloud Dataflow could chart sentiment towards the Brazilian football team dropping after the dubiously awarded penalty in their first World Cup game against Croatia.

However (there’s always a ‘however’ in ‘big data’), as if to contradict its own shift away from the existing big data standards, Google Capital then promptly led a new $110m (£64m) funding round for technology vendor MapR, which builds Hadoop systems for companies. That would tend to suggest a commitment to the status quo.

Google’s intentions are, as ever, a mystery to anyone but itself. It could be that the Cloud Dataflow launch is an attempt to phase out open-source Hadoop in favour of something Google can retain greater control over – big data didn’t seem such a big business opportunity when it let MapReduce out into the world. Or it could simply be that its new offering will be a complement to what already exists, ‘with added real-time’.

It’s at times like this that it becomes apparent why marketers and IT professionals have such a symbiotic relationship today. As a marketer, you’ll know just how valuable it could be to perform automated real-time analysis on the data you use, while your IT colleagues will be the ones to tell you how much it’s going to cost and whether the technology is worth the hype.

Is yours the kind of brand that would benefit from seeing, up to the second, how people are feeling and timing your messages to take advantage? If so, you should at least be aware of what’s becoming possible.

Latest from Marketing Week


Access Marketing Week’s wealth of insight, analysis and opinion that will help you do your job better.

Register and receive the best content from the only UK title 100% dedicated to serving marketers' needs.

We’ll ask you just a few questions about what you do and where you work. The more we know about our visitors, the better and more relevant content we can provide for them. And, yes, knowing our audience better helps us find commercial partners too. Don't worry, we won't share your information with other parties, unless you give us permission to do so.

Register now


Our award winning editorial team (PPA Digital Brand of the Year) ask the big questions about the biggest issues on everything from strategy through to execution to help you navigate the fast moving modern marketing landscape.


From the opportunities and challenges of emerging technology to the need for greater effectiveness, from the challenge of measurement to building a marketing team fit for the future, we are your guide.


Information, inspiration and advice from the marketing world and beyond that will help you develop as a marketer and as a leader.

Having problems?

Contact us on +44 (0)20 7292 3703 or email

If you are looking for our Jobs site, please click here