Laboratory (CSAIL), has also been a champion of Big Data for well over a decade. He believes that for most large companies, Big Data is less about managing the “volume” of data they have, and much more about integrating the wide “variety” of data sources that are available to them – which can include data from legacy transaction systems, behavioral data sources, structured and unstructured data, and all sizes of data sets.9 Stonebraker has estimated that corporations manage to capture only a small fraction of their data for analysis. His focus is on expanding the sources and varieties of data that companies can bring under management.
The Transformational Impact of Big Data
By the fall of 2012, Brynjolfson and McAfee had published their Harvard Business Review article, and Mayer-Schönberger and Cukier their Big Data book. Quoting management gurus W. Edwards Deming and Peter Drucker, Brynjolfson and McAfee noted that “You can't manage what you don't measure” and continued, “Simply put, because of Big Data, managers can measure, and hence know, radically more about their business, and directly translate that knowledge into improved decision making and performance.”10 Big Data was being widely embraced.
The National Bureau of Economic Research published a 2013 study, “The Data Revolution and Economic Analysis,” which begins with a 2010 quote from Eric Schmidt, former CEO of Google: “There was five exabytes of information created between the dawn of civilization through 2003, but that much information is now created every two days, and the pace is increasing.”11 The NBER report continued, “Many believe that ‘big data’ will transform business, government, and other aspects of the economy.” The Association for Supply Chain Management published a 2015 research study, “Exploring the Big Data Revolution.”12 In 2018, BizEd featured a story, “B-Schools Respond to the Data Revolution.”13 The common theme among these articles, studies, and books continued to be the revolutionary quality of Big Data.
The embrace of Big Data continued to accelerate, but an inevitable reaction, and counterrevolution, was on the horizon. By the latter part of the decade, the Big Data pushback was well underway. The Wall Street Journal had concluded back in early 2016 that Big Data was no longer the hot topic. The reaction had started to build force. In February 2019, Inside Digital Health ran a story, “Experts: Big Data ‘Revolution’ Won't Materialize Until Healthcare Fixes 1 Big Problem.”14 In July 2019, the Washington Post published an article, “If Baseball Is Any Indication, the Big Data Revolution Is Over,” commenting, “The data revolution has largely disappointed – and we shouldn't be surprised.”15 By November 2020, I had published my own article in Forbes, “The ‘Failure’ of Big Data,”16 which weighed the accomplishments of Big Data balanced against the ambitious prognostications of its earliest proponents. The reality had settled in. Big Data had brought transformative change in some quarters, but in other quarters it continued to be a case of business as usual or a time for reexamination.
Big Data was by now well-established, but the record of accomplishment was checkered, as organizations sought to overcome some basic misunderstandings that were common to many. It became a time to reflect upon and revisit just what it was that has made Big Data distinct and different from all that preceded it:
1 Big Data is not just about massive data sets. Big Data is broad and encompassing. It can refer to all data sources, large or small, structured data or unstructured data, and legacy or new sources, such as social media. Although Big Data initially referred to a set of data management technologies, such as Hadoop, that were first employed by social media companies such as Google, Facebook, Yahoo!, and others to enable the processing of massive volumes of information in a timely fashion, Big Data approaches and thinking can be applied to all data, regardless of size or other characteristics.
2 Great insights can come in small packages. Over the years, I have sometimes been told by senior corporate executives that they did not have a Big Data opportunity or need because they were not focusing on social media data, unstructured data, or massive data sets. Tom Davenport said it well in a 2012 Harvard Business Review article, “Even Small Data Can Improve Your Organization's Judgment.”17
3 Big Data brings business value. Tom Davenport remarked in his book Big Data @ Work: Dispelling the Myths, Uncovering the Opportunities, “I assumed [Big Data] was just another example of vendor, consultant, and technology analyst hype. I discovered I was wrong to be skeptical after I began doing research on the topic.” Davenport continued, “Big Data is such a broad business resource that it is sometimes difficult to envision all the ways that it can affect an organization and an industry.”18
4 Big companies are dealing with legacy data environments. While much of the discussion about Big Data has focused on the benefits and opportunities that result from bringing in new sources of data, including social media, sensor, and visual data, most of the action and investment is among mainstream corporations that are focused on integrating information from traditional legacy environments.
5 Big Data was embraced by the mainstream. As Big Data was absorbed into the mainstream, it has evolved from something that started as experimental and was largely restricted to discovery and sandbox activities that were relegated to research groups, which were isolated and not integrated into core business and technology processes. This is no longer the case for mainstream corporations. Big Data approaches have been nearly universally adopted over the course of the past decade. Big Data has been ingested into the mainstream and is here to stay.
The Business Value of Big Data
A few years after the initial breakthrough of Big Data, Silicon Valley Bank (SVB) held a Big Data Summit to highlight some of the challenges facing mainstream corporations as they struggled to seize an advantage from the opportunity presented by Big Data. SVB highlighted what they perceived as a “discrepancy in maturation” in Big Data capabilities based on the selected vertical markets or industries that they analyzed.
Referencing statistics drawn from the U.S. Department of Commerce, Gartner, and additional sources, SVB concluded that Big Data maturity is highest in smaller-sized markets, like advertising and travel, and is at an early stage or nascent in the large economic markets, like financial services and healthcare. They cited two significant impediments to Big Data spending in these markets: level of regulatory oversight and ease of data capture.
Despite sizable investments at the time in Big Data from some large financial services and healthcare companies, Silicon Valley Bank concluded that the Big Data industry was at the tip of the iceberg (“nascent”) in terms of future investment opportunities. SVB foresaw a future driven by Big Data investment in “high friction, high value” industries. Chris Parsons, a speaker at the SVB event, and who served as vice president, Big Data Strategy, for AT&T from 2014 to 2019, related that “the use cases may not always be sexy,” but the “business impact can be enormous.”19 AT&T had recently opened a dedicated Big Data center of excellence in Plano, Texas, with the goal of bringing speed and agility to its business capabilities and processes. For AT&T, Big Data meant dealing with 10 million columns of structured data for 62,000
