Making good decisions

Currently, decisions leaders make are intuitive guesses. I argue this does not have to be the case. Instead, various tools are emerging that can reduce the uncertainty in decisions. For example, prediction markets can yield much more precise estimations of risks than individual experts. However, prediction markets are expensive to run and burdensome to participate in. Nevertheless, AI emulations can be used to replace human participants in prediction markets, paving the way to synthetic, precise probability estimates for all external events.

Every decision is a decision towards the future. But the future is unknown; there cannot be knowledge about the future. Thus, leaders lead by making educated and uneducated guesses about the future. Could the future be more clear to us if we would use the digital platforms, data and AI more efficiently?

According to a McKinsey survey, only 20% of leader believe their organisation is good at making decisions. If we think that decisions fall short on benefiting from the vast amount of information available, we should examine the decision-making infrastructures and make changes. In an another survey, by Oracle, 72% of leaders admit the sheer volume of data and their lack of trust in data prohibits them from making decisions. If we do not like the current decision-making infrastructures, we need to show why it leads to unacceptable or underperforming decisions.

How good are we at making decisions?

First, the good news. In a suitable setting where people try their best, we can collectively make decisions that significantly deviate from random. For example, a study on prediction markets demonstrated that people can predict the outcomes of successful replication of scientific studies with 73% accuracy.

A prediction market is a tool where a large number of people participate in gambling regarding a future-relevant phenomena. Such gambling can include questions about probability (“What do you estimate as the likelihood of peace in Ukraine during the year 2024?”), date (“When does the Russian attack to Ukraine end?”) and a number (“How many Russian tanks are visually confirmed to be destroyed during 2024?”). Participants can change their answer when new information occurs, but this might have a cost. In any case, the prediction usually becomes more precise in time.

The power of the prediction markets is that while we cannot know how the future turns out, collectively we have a lot of information about path dependencies, motivations, structures, delays, events and other things and this allows us to plan and act regarding the future with nonrandom accuracy.

Prediction markets are one of the best known infrastructures for producing decision-improving information about an important external phenomena, event, number or date. Probability of an external event is often not easy to grasp or act upon using time series analysis, but simultaneously a decision needs to be made (and not making a decision counts as a decision). In these situations, analysts and decision-makers employ judgemental forecasting. Judgmental forecasting means making predictions about future events based on subjective opinions, insights, and intuition rather than on purely quantitative data or mathematical models. This method leverages human judgement, expertise, and experience to forecast outcomes, especially in situations where hard data may be scarce, unreliable, or inapplicable.

In fact, most foresight (but not all of it) belongs to this category of predicting probabilities of or speculating about external events with judgemental forecasting.

All in all, a lot of work is done to support decision-makers. Furthermore, we know that there are approaches such as the prediction markets that could be used to improve decision-making further.

Then the bad news. While prediction markets are the best way to synthesise insights about the future, unfortunately they are rarely used for this as they are slow, expensive and burdensome. Likewise, many other tools, heuristics, frameworks, dashboards and software are available, but rarely used. Instead, most leaders make decisions largely through an unconscious process of pattern recognition and emotional tagging, even ignoring the best practices that they often are aware of. In the current leadership paradigm, the decision-making system is thought to be placed within the heads of the individuals. As an outcome, large companies pay their CEOs massive salaries. For a similar reason, most interventions within the decision-making system are focusing on the individuals, for example, on biases that these leaders might possess.

Bias-based paradigm of improving decisions suggests that decision-making is necessarily an introspective and implicit act: a cognitive operation within the mind of a decision-making. However, there’s more to the decisions than monitoring and training the output of the leader introspectively. Bias in decisions is an outcome of the decision-making infrastructure, where individuals are left alone in crafting complex synthesis from conflicting and scattered data.

Instead of aiming to reduce bias in individuals, I suggest improving and implementing the digital infrastructures of knowledge synthesis, storage, retrieval and analysis.

A better decision-making infrastructure

Many organisations have excellent strategy departments, analysts, and/or foresight capabilities. The very same organisations can have a good decision-making culture or even an exceptionally talented leader. However, synthesis from analysts is rarely used to guide decisions, and analysis, especially foresight, is seldom affected by decision-making needs.

In Finland, I have witnessed this first hand when working with top governmental and private sector organisations on their foresight: they typically have excellent foresight people and departments, but these people have a rather distant connection to the decisions made within the organisation. Often, they have two outspoken purposes in the organisation in 1) preparing the organisation’s strategy, at most once a year, and 2) in creating a “foresight culture” in the organisation. At worst, the former purpose leads foresight teams to produce harmless common knowledge, while the latter purpose is a keyword for “do not disturb the day to day operations”.

Few would disagree that decision-making could benefit more directly from the work of the analysts, if only the insights would reach the decision-maker at the right time and format.

Usually, the problem of disconnect between analysts and decision-makers is attributed to organisational silos. However, while such ‘horizontal’ silos exist in most organisations, I would suggest that the problem is more fundamental. On top of ‘horizontal silo between organisational departments, there exists also a ‘vertical’ silo between different kinds of knowledge needed from decision-makers and provided by the analysts. A politician cannot comprehensively turn the result of a scientific article into a policy action; a journalist cannot fully explain the background situation of a political decision to the audience.

Furthermore, because companies and organisations employ ‘analysts’, leaders tend to ask for analysis, when they in fact need a synthesis! A synthesis, a description of what a phenomenon, event or decision is about, is in fact always required to make a good decision: because analysts provide them with analysis, leaders have to synthesise themselves. I claim that this is not a good practice, as it leads to assigning random value to different kinds of information and different pieces of analysis (aka ‘bias’) . Instead, the decision-making infrastructure should nudge analysts to produce synthesis directly.

Aiming towards a synthesis already outside the head of the decision-maker helps to focus to three tasks that any knowledge-based decision-making infrastructure has to do:

• Gathering information
• Providing advice
• Making an action

Information gathering supports advice and advice supports action. We can look at each of these three phases individually, and when we do, we start to realise why decision-making is so difficult and perhaps also begin to realise how we can improve it.

Regarding gathering information, no one knows what kind of information an analyst should use. There is no plausible meta-theory on which kind of information should be prioritised in which situation. There are no heuristics demonstrated to show successfully which kind of knowledge works for each different situation. Only thing that can be said is that typically, good advice combines different kinds of information.

Different categories of information according to Geoff Mulgan include:

• Statistical knowledge (unemployment rises due to sanctions)
• Policy knowledge (what sanctions can be applied)
• Scientific knowledge (antibody testing is possible)
• Disciplinary knowledge (sociology or psychology on patterns of community cohesion)
• Professional knowledge (how wheat can be replaced in nutrition)
• Public opinion (quantitative poll data and qualitative data)
• Practitioner views and insights (police experience in handling breaches of the new rules)
• Political knowledge (when parliament might revolt concerning a topic)
• Legal knowledge (what actions might be subject to judicial review)
• Implementation knowledge (understanding the capabilities of different parts of government to perform different tasks)
• Economic knowledge (which sectors are likely to contract most)
• ‘Classic’ intelligence (how global organised crime might be exploiting the crisis)
• Ethical knowledge about what’s right (priority of gasoline rationing)
• Technical and engineering knowledge (how to use LLMs safely)
• Futures knowledge (foresight, simulations and scenarios, e.g. recovery speed)
• Knowledge from lived experience (the testimony and experiences, usually shared as stories, for example about experiences of a crisis response)

Looking at the list, it is obvious that the job of the analyst is not to filter and frame any piece of information for the decision-maker. Instead, any good advice for decision-makers need to synthesise the often-contradictory signals coming from the different kinds of knowledge. To work as an analyst means to synthesise.

“Good advice synthesises information.”

Regarding providing advice, there are many ways to synthesise. Geoff Mulgan lists the following:

• Synthesise downwards: put multiple things into a single metric (Cost benefit analysis, QALY)
• Synthesise upwards: a theory that explains many things at once (Evolutionary theory)
• Synthesise forwards: decide on strategy (preparation plan, implementation plan)
• Synthesise backwards: make sense of historical patterns
• Synthesise with analogy (“the earth is a single organism”, “a pandemic is like a war”, “the spread of an idea is like a virus”)
• Synthesise through a heuristic: suggest a simple rule that can work most of the time (a monetary policy target, “start-ups have to focus on cash flow”

Of course, a synthesis can also be a combination of these different types .A learning from this list is that analysts should make sure they are working towards a right kind of synthesis.

Regarding making an action, a better decision-making infrastructure would synthesise advice from multiple sources of sometimes conflicting information and provide the advice in a format that benefits decision-making directly.

A best possible decision-making infrastructure would thus

First, offer the decision-maker complete awareness of their organisation’s surroundings, and

Second, offer the decision-maker an actionable predictive awareness of their organisations’ surroundings.

Konsensus.me is a solution for the first task. Souls, an automatic risk probability estimation system we are developing on top of Konsensus.me, is a solution for the second task.

Complete awareness of the operational environment using Konsensus.me

Konsensus.me is an AI tool for peer-verified constantly updated situation analysis, providing decision-makers with strategic intelligence about important phenomena. It provides the users only the relevant information by scanning a large corpus of legal texts, research and reports and constantly monitors the news, automatically identifying new information that has an impact.

When a new insight is discovered, Konsensus.me does more than just deliver the user the link: it also suggests how the insight might change their understanding about the phenomena that matters. Experts verify these suggestions with one click, simultaneously teaching the AI.

Complete predictive awareness

Future is big; there is a lot that can be said about the future. To have control over their future, leaders should aim to synthesise most advice downwards to actionable metrics such as probabilities regarding key risks and events. The most precise way to produce such information is running a prediction market with external participants.

The problem is that prediction markets are very expensive to host and very demanding on the participants. Thus, only a few organisations, namely Google (source) and most likely some Intelligence Agencies, can afford to run prediction markets at reasonable scale.

The problem could be fixed if it would be possible to emulate prediction market participants with LLMs. Initially, it was proven that LLMs such as GPT4 cannot do prognosis; as standalone products, they are not useful in prediction markets. To be precise, even the most advanced models are completely random in answering prediction market questions — as good as monkeys. Questions about the future are difficult for language models because there cannot be knowledge about the future in their training material; thus, the LLM as such is forced to hallucinate an answer to the foresight question, producing poor results.

However, LLMs can greatly help in prediction markets — when they are used correctly. Simply prompting LLMs to start their estimation reasoning with a plausible base rate estimate improved its performance significantly. A more complex structure of LLMs that combines LLM-based retrieval, reasoning and aggregation systems separately has very recently been proven to reach close to human performance or even surpass it.

I have a bad habit of burying the lede, but here it is: that last research result is completely world-changing. Demonstrating analytically that it is possible to reach human-level prognosis with digital tools, it opens up a possibility to maintain a shared, explicit, constant, cheap, and more precise understanding on the probabilities of future events.

A tool that does that has a massive number of use cases. It can be used to instantly measure impact of any action to any precise context (!); or to remove the need for market mechanisms in at least some intangible insurance and information contexts (!). At the very least, I cannot come up with a reason for an organisation to rely on future scenarios if such synthetic prognosis is feasible, so at least futurists should pay attention to this.

I’m not only talking the talk here. We are in the process of replicating and building on the results by Halawi et al (2024), using Konsensus.me as the retrieval system and our LLM-emulation based synthetic emulation prediction prototype “Souls” as a reasoning system. We consider it likely that we reach human level performance in constrained foresight contexts and can prove it during the spring.

Better decisions

If we can prove that it is feasible to generate better-than-human risk estimates synthetically at scale, we can offer decision-makers precise and always up-to-date risk estimates on many if not most external risks. If a decision-maker takes these estimates into account in their decisions, they gain a plausible explanation post ante to questions regarding their unsuccessful decisions: if they used our system and if our proof of its abilities is undisputed, a leader can show the decision was optimal even if incorrect based on the available estimate at the time.

I would assume using such a system is tempting to many decision-makers.