The McKinsey report was prescient in its observation that “Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers.” To me, this was the distinguishing insight. From this point forward, data would no longer be the purview of “a few data-oriented managers” but rather would become the purview of “leaders in every sector.” Data would be open to the masses. Call this the democratization of data. The McKinsey report went on to characterize the dawning era of Big Data as heralding “new waves of productivity growth, innovation, and consumer surplus.” Big Data, like the Internet before it, was destined to change the world. Notably, the McKinsey report did include one important caveat, or small print disclaimer, that the great promises and grand benefits of Big Data would transform business “as long as the right policies and enablers are in place.”
The term Big Data has served to popularize data in a way that decades of data work had never previously achieved, by making it simple, accessible, understood, and big and important. The sudden ascension of Big Data led to my own two-year stint as a Wall Street Journal guest columnist during 2014–2015, writing monthly columns on Big Data and its industry-transforming power and potential. The attentions of Fortune 1000 companies were now engaged at the highest level, compelling the interest of the corporate executive committee and board of directors. This was the moment that leading global corporations embraced data as a critical business asset, and recognized that to compete in the twenty-first century they must possess a data strategy and data leadership.
The advent of Big Data was met by an enthusiastic embrace by the venture capital, private equity, and global investment communities as well. By November 2011, the prominent venture capital firm Accel Partners, an early investor in Facebook, had announced the launch of a “$100M Fund to Invest in Disruptive Big Data Companies.” Accel partner Ping Li noted, “this Big Data wave will usher in a new era of multi-billion software companies that solve existing problems differently or new problems previously unthinkable.”4 Accel also announced the launch of a Big Data conference to “network and drive deeper discussion on technology trends in the space.”
The Big Data Value Proposition
On a picture-perfect May morning in 2012 on the campus of Stanford University in Palo Alto, California, I attended the inaugural Big Data conference at the invitation of Accel Partners. This event represented a milestone in the history of Big Data. Accel Partners had emerged as the hottest venture firm in Silicon Valley on the heels of their successful investment in Facebook. The event was billed as a “who's who” of the Silicon Valley technology elite. A parade of notable speakers, including Andy Bechtolsheim, co-founder of Sun Microsystems and Doug Cutting, originator of Hadoop and chief architect at Cloudera, the latest Accel investment, followed one another, each extolling the “revolutionary” potential of Big Data. The speakers set forth grand claims for the promise and potential of Big Data. “Big Data was the next wave in technology innovation,” remarked one speaker.5 “Big Data would change how the world made use of information,” stated another. “Big Data would enable insights that would change society,” commented a third.
Each speaker extolled the technological underpinnings that made Big Data truly compelling, which was the notion that you could simply take your data and “load and go,” thereby shortcutting the massive effort of data preparation and data engineering that was the bane of corporate existence, comprising 80% of data activities. This specifically pertained to the tremendous time and effort required to transform data (“dirty data”) into a usable asset that has meaningful business value. Whole companies and an entire industry had been built to respond to this need. It was the ongoing lament of many a data analyst who complained that they spent “20% of our time on data analysis, but 80% of our time on accessing and preparing the data.”
The big attraction and promise of Big Data for many corporations was the ability to bypass the hundreds of hours of upfront data engineering to access the data much sooner and more easily, for purposes of analysis and putting this information to good use. Anyone who has ever worked in the corporate world knows the painful refrain to how long it takes to answer a new business question that requires adding a new data source: “Fifteen months and five million dollars.” Senior business executives were resigned to a state in which getting value out of data quickly was not something that they could expect to see in their business lifetimes. Then, a cadre of engineers, data experts, and venture investors were heralding a breakthrough. Data would be liberated from the tyranny of the data gatekeepers. Data could now be made available quicker and faster to all.
The radical implication of the load-and-go notion was that data users would no longer have to wait through the long and arduous processes of data engineering that had long hamstrung and thwarted the ambitions of data analysts. Speed would be realized through the ability to shorten the cycle from data access to analytical results. An analogy would be the reduction in travel time resulting from the railroad versus the horse and buggy, or air transport versus travel by land or sea. Big Data would make it possible to get more done for less by lowering the cost of managing data by reducing the need for costly upfront data preparation and data engineering, the dreaded 80% of time and effort.
The Implications of Big Data for Large Companies
Long saddled with disparate sources of legacy data, corporations could for the first time successfully integrate these sources due to the cost and speed advantages resulting from Big Data technologies. Corporations could access new sources of information, such as unstructured data sources including documents, text, and pictures, and behavioral data captured through social media channels. The result was a growing sophistication in the data and analytics capabilities of mainstream companies.
Jonathan Bennett is a long-time top executive, EVP, CFO, and business head at The Hartford, a $26 billion insurance and investment firm founded in 1810 with a long history of actuarial analysis, where data has always mattered. Mr. Bennett has maintained a clear-eyed view of both the opportunity and the challenge represented by Big Data. In 2013, he noted that keeping a focus on cost and the benefits of better managing data “is just as important as breaking into new Big Data opportunities. If we can figure it out, cost reductions from the former will help fund expansion in the latter.”6
Big Data did not make traditional data management methods disappear overnight. Change seldom comes quickly. The apostles of traditional data management approaches battled back and demonstrated that data transformation is not quite always as simple as just “load and go.” Any transformation implies change. This meant required changes to business processes, technology tools, employee skill sets, and, most especially, to human mindsets. Although some data engineering has been eliminated or reduced, and Big Data approaches have dramatically reduced the costs of data management, data still needs to be standardized, data quality maintained, and access provided to analyst communities. Data management will continue to be an evolutionary process for companies.
One year after their Stanford University conference, Accel would launch a second $100 million fund, with Accel partner Jake Flomenberg, proclaiming, “Over the past few years, we've focused a tremendous amount of attention on what people like to call the ‘three Vs’ of big data: variety, volume, and velocity. We now believe there is a fourth V, which is end user value, and that hasn't been addressed to the same extent.”7 Accel's Ping Li added, “We are seeing an accelerated rate of innovation in big data, with the newest generation of entrepreneurs re-imagining ways to extract the most value out of big data and fundamentally change the way we work and process information.”8
Michael Stonebraker, a pioneer in the field of data management and the 2014 recipient of the Turing Award, the
