without the need to purchase or build large supercomputers. This means that any business, government body, or indeed anyone can now use Big Data to improve their decision-making.
Especially powerful is our ability to analyse so called ‘unstructured data’ (more on this in Chapter 3). Basically, unstructured data is the data we can't easily store and index in traditional formats or databases and includes email conversations, social media posts, video content, photos, voice recordings, sounds, etc. Combining this messy and complex data with other more traditional data is where a lot of the value lies. Many companies are starting to use Big Data analytics to complement their traditional data analysis in order to get richer and improved insights and make smarter decisions.
In effect what Big Data should really stand for is SMART Data and whilst I think the term Big Data will disappear in time, the increasing production and use of SMART Data is definitely here to stay.
Who is using Big Data?
The big players in the space, including Amazon, Google, Walmart, and Facebook, are already making a splash. Walmart, for example, handles more than a million customer transactions each hour and imports those into databases estimated to contain more than 2.5 petabytes of data.4 The company is now able to combine data from a variety of sources including customers' past purchases and their mobile phone location data, Walmart internal stock control records, social media and information from external sources such as the weather, and initiate tailored sales promotions. For example, if you have bought any BBQ-related goods from Walmart, happen to be within a 3 mile radius of a Walmart store that has the BBQ cleaner in stock, and the weather is sunny, you might receive a voucher for money off a BBQ cleaner delivered to your smart phone!
In another example a client of mine, a leading telecom company, is using Big Data analytics to predict customer satisfaction and potential customer churn. Based on phone and text patterns as well as social media analytics, the company was able to classify customers into different categories. The analytics showed that people in one specific customer category were much more likely to cancel their contract and move to a competitor. This extremely useful information now helps the company closely monitor the satisfaction levels of these customers and prioritize actions that will prevent them from leaving and keep them happy.
Even mid-tier cars today have about 40 microprocessors that measure performance. These electronics usually account for about one-third of the cost of a new car. Of course, all this data that is being generated, collected and analysed by the car manufacturers offer them significant competitive advantages. One car maker working with an external analytics company noticed that a sensor in the fuel tank made by a German supplier was not working well at all. The manufacturer could have told the supplier and asked them to fix it but then the improvement would have been passed on to other car manufacturers that use that supplier. So instead the manufacturer invented a software patch that fixed the issue, received a patent on the fix and sold the patent to the supplier.5
Big Data is changing the very nature of business, from manufacturing to healthcare to retail to agriculture and beyond. The rate that data is and can be collected on every conceivable activity means that there are increasing opportunities to fine-tune procedures and operations to squeeze out every last drop of efficiency.
How companies are using Big Data
Different industries have responded to the call in different ways. Retail and sales are seeking to collect as much information about their customers' lives as possible so as to fulfil their changing needs more effectively. Manufacturing are seeking to streamline operations. Equipment calibration settings can be recorded and refined, and product storage environments monitored to determine the optimum conditions that lead to minimum spoilage and waste.
For global companies this can mean collecting and analysing data from plants across the world, allowing minor variances to be studied and their results understood.
In 2013, for example, pharmaceutical giants Merck used analysis to dramatically cut the amount of waste caused by variance in manufacturing environment conditions. It took three months and involved 15 billion calculations on individual production data from 5.5 million vaccine batches. This allowed them to discover the optimum conditions during the fermentation process, and should greatly increase their yield, once the FDA has approved the proposed changes to the manufacturing process.
In the automotive industry a 2014 report by the Centre for Automotive Research stated that advances made possible through advanced IT solutions and Big Data represented ‘an engine of innovation’. The report highlighted the growing complexity of cars and the industry as the biggest challenge faced by automotive manufacturers.
The efficiency of every machine – and human – involved in the manufacturing process can be recorded so companies know what is working, and can make improvements where they are needed.
And in agriculture, data analysis is helping the industry meet the challenge of increasing the world's food production by 60 %, as forecasters have said will be necessary by 2050 due to the growing population. Tractor and agricultural machinery manufacturer, John Deere, already fits sensors to its machinery. The data that is available to the farmers via its myjohndeere.com and Farmsight services helps them to establish optimum conditions for their crops. Plus the data is also used by John Deere to forecast demand for spare parts.
Of course, in business once a product has been grown or manufactured it needs to be sold and distributed. The petabytes of customer data, including you and me, already gathered by big retailers tells them who will want to buy what, where and when. Amazon, for example, uses its S3 system to keep track of millions of stock items across dozens of warehouses and distribution centres scattered around the globe. Operatives can track deliveries in real-time to see what is where, and where it should be going.
At the point of sale, retailers can use data to determine where stock should be displayed, which stores will sell most of which particular product and track customer movements around stores. Loyalty cards are not new but ever more sophisticated analysis of customer habits will lead to an increase with which retailers can predict what you will buy. This has advanced to the point where Amazon believes it will soon be able to predict what you will buy accurately enough to despatch it toward you before you have even bought it!
The connectivity that is now possible is also changing business. In 2014 Cisco announced a $150 million fund for start-ups working on improving integration between the virtual and physical world. For a business, the ability to have its production, stock control, distribution and security systems all connected and talking to each other will mean greater efficiency and less waste. GE refers to this convergence of data and machinery as the ‘Industrial Internet’, and claims it can save global industry £150 billion in wastage.
Every area of industry is learning to reap the benefits of Big Data analysis, and it looks certain that finding innovative methods of gathering, recording and analysing data is going to play a big part of business in the foreseeable future.
Even something as subjective and ‘human’ as Human Resources is being transformed by Big Data and analytics. Finding and keeping the right people is a major bugbear for most businesses. Talent management is fraught with challenges and the cost of failed management and leadership is enormous. It is estimated that the average cost of executive failure is $2.7 million.6 Published estimates into the extent of poor leadership range from 33%7 to 67 %.8 In other words between one- and two-thirds of all current leaders will fail in their role.
But it's not just a financial cost. Unsuccessful executive appointments alone incur significant hidden costs, which can include lost opportunities,
SAS Whitepaper (2012) Big Data Meets Big Data Analytics: Three Key Technologies for Extracting Real-Time Business Value from the Big Data That Threatens to Overwhelm Traditional Computing Architectures.
Mayer-Schonberger, V. and Cukier, K. (2013)
Smart, B. D. (1999)
Sorcher, M. (1985)
Hogan, R., and Hogan, J. (2001) Assessing leadership: A view of the dark side.
