Richard M. Adler
The Shortcomings of BI and Analytics for Critical Decisions
The media and trade press are awash in claims about a veritable golden age of decision support, driven by big data, business intelligence (BI), analytics, and artificial intelligence technologies such as Machine Learning (ML). The common theme is that companies which embrace these powerful tools can leverage their vast data stores to make better decisions that will benefit all of their stakeholders. Discounting marketing hype, it is undeniable that tremendous advances have been made in decision sciences, software technologies, and computing power in recent decades. That said, this rising tide doesn’t raise all decision-making boats equally.
BI tools aggregate, summarize, and help navigate historical and current business data sets. Analytic solutions help identify correlations, trends, and other patterns in data and project them into the near-term future. These capabilities clearly enhance the front-end of the decision-making process, which includes recognizing emerging problems or opportunities, data gathering, and situational understanding (aka sensemaking).
However, the value of these contributions vary in importance depending on the type of decision. Consider business decisions as forming a spectrum defined by duration and scope. Tactical and immediate operational decisions, such as maintaining inventories, lie at one end of the spectrum while the other end consists of critical decisions that play out over years or decades and shape the well-being and even survival of the enterprise. Critical decisions include mergers, changing core business strategies, and costly, high-risk projects such as developing new drugs or oil fields.
On the short-term side of the decision spectrum, as advertised, technologies such as BI, analytics, and ML undoubtedly play a central role. Consider operational decisions to replenish product inventories or production inputs such as raw materials or components. These decisions are routine, short-term, well-bounded, and occur frequently. Recognizing the need for action in these constrained contexts is an accounting exercise in tracking inventory and sales. Decision framing consists of balancing two operational objectives—minimizing inventory carrying costs while ensuring satisfaction of anticipated demand. Decision options consist of calculations as to what products or materials to re-order, in what quantities, and when. These options can be formulated by using forecasting tools, business rules, optimization algorithms or ML. Thus, basic BI reports and analytics on historical patterns and current trends for stocking and sales patterns are largely sufficient by themselves to trigger and guide ordering decisions.
By contrast, as the scope of decisions expands and their time horizons lengthen, BI and analytics lose their exclusivity as inputs to decision-making. Their importance is diluted because situational understanding for critical decisions depends on factors beyond operational data and patterns such as information about broader market, economic, and sociopolitical conditions. Common gaps in big data and BI include customer behaviors, intentions of competitors, and contingencies such as future disruptive events, trends, and forces. This information is often qualitative rather than numeric, which poses challenges for quantitative analytic methods. Data doesn’t suffice to make sense of situations; instead, expert judgment is required to interpret that data, pose questions, fill in gaps, and form “the big picture.” As decisions shift toward the critical end of the spectrum, BI and analytics transition to a reduced role of providing partial inputs to decision-makers.
As the scope and time horizons for decisions increase, the utility of BI and analytics as direct guides to action diminishes as well. Recognizing the need to act on a critical level hinges not simply on data, but on value judgments: what is good about a company’s current state, what is bad or dangerous, what level of response is necessary, and what is the urgency to act? Decision framing is no longer confined to one or several quantitative metrics like inventory levels and stock-out costs. It now depends on complicated trade-offs between metrics that often compete rather than align (e.g., output, cost, ROI, quality, time-to-market). Framing must also accommodate competing interests of stakeholder groups and business values, norms, and laws that constrain strategic actions.
Formulating and evaluating options to act are also far more complicated for critical decisions than for operational ones. Predictive analytics project trends and associations into the future. Some of these patterns correlate business actions directly to effects, dictating many types of well-bounded decisions. Suppose, for instance, that the manufacturing yield for a product is known to be 90%; one out of ten units fails to meet specifications. This statistic is more or less sufficient to dictate production decisions to satisfy demand objectives: on average, 112 pieces must be processed to make 100 good units (i.e., 100 / 0.9). By contrast, critical decisions are much more vulnerable than operational ones to situational changes because of their broader scope and extended durations. For instance, contingent events or changing trends or forces are more likely to occur as time horizons increase, disrupting situational conditions and increasing the likelihood that customers and competitors will deviate from their historical behavior patterns. Extended time horizons equate to uncertainty, which lessens the likelihood that the future will continue to resemble the past and future. Predictive analytics presuppose situational continuity. If that assumption is suspect, then so is the reliability of analytics in projecting decision outcomes and directing action.
Critical decisions generally have multi-faceted and interdependent goals and objectives. Ensuring that a decision will achieve these targets requires more than setting direction or defining a vision of the end state (e.g., a completed merger or a new enterprise software package integrated and running smoothly). A specification of the means for carrying out a decision option is necessary to evaluate its feasibility and its associated benefits and risks. Such implementation plans, even at a high level, should allocate resources according to workflows and schedules, and account for obvious contingencies and other risks. Predictive analytics offer very little support to this phase of critical decision-making, which requires project design expertise and creativity rather than statistical patterns.
Many critical decisions must be judged not only on their outcomes, but also how they are achieved over time, or path dependencies. For instance, two strategies for managing enterprise risks may reduce risk by the same amount over a fixed interval of time, but one reduces risk exposure more quickly than the other. All other things being equal, the path-dependent metric of speed to reduce risk breaks this tie. However, path-dependent performance metrics often change in ways that don’t average out cleanly over time. They may accumulate non-linearly, through compounding (or viral growth), or unevenly and piecemeal, in projects that play out in stages (e.g., design, development or acquisition, and roll-out). This means that critical decision-makers require visibility into how path-dependent metrics accumulate over time to make proper trade-offs. Predictive analytics are poorly-suited for this purpose because they forecast discrete end-states, much like the “after” snapshot in ads for acne or weight loss products. By contrast, simulation technologies generate the equivalent of movies to project the behavior of path-dependent metrics continuously over time. In addition, BI and analytics fail to capture the nuances of stakeholder preferences and the dynamics of adaptive behaviors such as strategic competition. Once again, businesses must marshal other decision science techniques to understand the influence of these factors on decision outcomes (e.g., agent-based models, utility theory, and game theory).
Finally, ML algorithms require large data sets to train them to detect useful patterns, and are vulnerable to inadvertent sampling biases. However, critical decisions occur far less frequently than operational ones. They also tend to exhibit sensitivities to initial conditions and time dependencies in plans. ML algorithms don’t perform well for data sets with these properties.
In conclusion, BI and analytics provide significant support to operational decision-makers. But critical decisions are broader and more complex that operational decisions; they require broader (and more qualitative) information, expert judgments, and creativity to make sense of situations and to define alternative courses of action. They also require more decision support “horsepower” from simulations and other decision science techniques to evaluate options and their consequences. Thus, contrary to conventional wisdom, BI and analytics are far less instrumental in supporting critical decisions, providing valuable, but partial inputs at best.