is a result of the formal recognition of another truth too: no matter how far AI/ML has advanced, combining it with in-house business knowledge always makes for better business outcomes.
Ushering in The Great Revival: Institutional knowledge and human expertise
Two of the biggest casualties in traditional data mining are institutional knowledge and human expertise. Institutional knowledge is defined as the knowledge within an organization about its own business and customers that’s passed on from older workers and leaders to newer ones in informal and usually verbal person-to-person exchanges. Because much of it is stored in the minds of workers and executives, it’s supremely difficult to identify, retrieve, and digitalize for inclusion in a data set. Therefore, it’s often lost when a person with some of this knowledge retires, dies, changes jobs, or otherwise stops being an active part of the company. Without this key information, business decisions can be made in the wrong context for the situation and result in failure or undesirable consequences.
Human expertise works similarly: It’s the knowledge gained by an individual by way of education, intuitive intelligence, talent, accumulated skill sets, experience, exposure, incidents of failure and success, encounters with anomalies and repetitious events, and a myriad of unique circumstances over the span of a career or lifetime. This information, too, is difficult to digitalize and add to a database. Therefore, human expertise also tends to be lost to illness, retirement, job change, or death.
The cost to any organization of the loss of either institutional knowledge or human expertise can be enormous in terms of money value, company culture, and the shape of the organization’s competitive edge. These facts are not lost to many in business leadership and data science, which is spurring a revival in both valuing and capturing these deep wells of specialized and irreplaceable data.
Some think of it as a great revival as the pendulum swings from one extreme to the other. For example, disinterest in customers from a focus on profits alone has now swung to a near-fanatical interest in personalizing every customer encounter and ensuring a great customer experience for each individual. This swing comes from a renewed appreciation for the value of human expertise (in this case in sales and marketing) and in institutional knowledge of customers and operations with regard to improving profits. In other words, once data-driven process efficiencies had mostly or completely been realized, companies learned that profits cannot be separated from customers, as the latter begats the former. Hence the resumed interest in reselling to existing customers and retaining customer knowledge beyond basic financial transaction details.
After all, data sets are still huge. Even if you use and find value in only 10 percent of data, that’s still an awful lot of data to parse and analyze. AI is also faster than people at, well, everything. That’s an advantage that organizations want to maintain. Then there’s the need to automate tasks so that work gets done faster, more efficiently, and without the need to interrupt function, features, and the customer experience.
Where do all these factors come together in the decision intelligence effort? Well, that’s what you and your organization have to figure out for yourselves. Certainly, there are guidelines on how to do it as well as tools at your disposal that I present in greater detail later in this book. However, remember that the first part of bringing humans more fully into the decisioning roles is in deciding the particular blend of human versus machine processes that are needed.
Or, to put it another way — and in keeping with Kozyrkov’s earlier analogy — this is the part for the microwave chefs to work their magic in making the recipe. The job of the microwave builders is largely finished. You know you have the right recipe when you see the proof in the pudding, so to speak.
Chapter 3
Cryptic Patterns and Wild Guesses
IN THIS CHAPTER
Yahoo! put the first Hadoop cluster — arguably, the first truly successful distributed computing environment designed specifically for storing and analyzing huge amounts of unstructured data — into production back in 2006. It’s that date which, for most practical purposes, marks the onset of the big data gold rush and the hunt to discover unknown information buried in known data sets.
The results were largely perceived to be worth the effort and generally enlightening — even though most big data initiatives fail to this very day. Even so, the call for data driven businesses, to the chagrin of business leaders and managers everywhere, became the mantra in business and investment circles worldwide. Organizations were soon convinced that using data analytics meant the same thing as harvesting answers. The thinking was that the answers generated were perfect right out of the box and were produced by means far beyond mere human abilities. Gut instinct and human talent were summarily discounted and dismissed as little more than wild guesses. However, the reality was and is quite different, as analytics have limits, big data and AI projects have high fail rates, and business executives very often let their gut instincts override algorithm outcomes.
Fear of AI began to soar as people expected machine masters to leap from science fiction and rule the real world. But that’s a far cry from what has happened so far.
The notion that data analytics was somehow churning out answers at a record pace gave way to the broadening realization that what analytics actually was producing were insights. Humans were still needed to glean understanding and perhaps inspiration from those insights and to turn them into decisions and actions.
It turns out that machines aren’t the new masters of the human race, after all. And they don’t provide the final answers humans seek. But that’s more the fault of humans than the machines. People were so busy asking questions of the data that they forgot to look where the work was headed. Organizations often found themselves working in circles or solving problems that yielded no tangible benefits for the questioner.
Machines Make Human Mistakes, Too
People commonly believe that machines are unbiased and more perfect than humans. Data analytics, automation, and machine learning (referred to as AI by marketers everywhere) are often presented as though the machines are capable of sorting out the data and reaching a perfect and fair conclusion on their own.
This simply isn’t
