Explainable AI

A machine learning model can be extremely accurate and still be a mystery. It takes an input, produces a score or a label, and gives you no account of why. Explainable AI, usually shortened to XAI, is the set of concepts and techniques for closing that gap: turning a model's output into something a person can understand and check.

The problem it solves is trust under uncertainty. When a model denies a loan, flags a transaction as fraud, or recommends a treatment, someone has to answer for that decision — a developer debugging a failure, an auditor checking for bias, a regulator verifying compliance, or the person the decision affects. None of them can do that job from a number alone. They need a reason.

Why "accurate" is not "explainable"

Accuracy measures whether a model's predictions match reality. Explainability measures whether a human can understand how the model arrived at a given prediction. The two are independent. A simple linear regression is easy to explain and often less accurate; a large ensemble of trees or a deep neural network is often more accurate and, by default, opaque. That opacity is why practitioners call these black box models — models whose internal reasoning is hidden from the people who use their output, even when the code and weights are fully visible.

XAI does not make a black box model simpler. It builds a second, smaller model or computation on top of the first one, whose only job is to describe the first one's behavior in terms a person can follow.

Two ways to get an explanation

Some models are understandable by construction. A linear regression's coefficients, a shallow decision tree's branches, or a rule list's conditions are the explanation — there is no separate step. This is intrinsic interpretability: the model design itself is legible.

Most high-performing models are not built that way. For those, you reach for post-hoc explanation: methods applied after training that probe the model's behavior without changing it. Post-hoc methods split along two more axes:

• Model-specific vs. model-agnostic. A model-specific method uses the internal structure of one model family — reading a tree's splits, or computing gradients through a neural network. A model-agnostic method treats the model as a black box and works by observing inputs and outputs, so it applies to any model that produces predictions.
• Local vs. global. A local explanation accounts for one prediction: why did the model reject this specific loan application? A global explanation describes the model's behavior in aggregate: which features matter most across the whole portfolio? The two answer different questions, and a model can look very different up close than it does in aggregate.

The working techniques

Two model-agnostic methods dominate practice. LIME (Local Interpretable Model-Agnostic Explanations) explains one prediction by perturbing the input, observing how the model's output changes nearby, and fitting a simple, interpretable model — usually linear — to that local neighborhood. SHAP (SHapley Additive exPlanations) takes a different route: it borrows the Shapley value from cooperative game theory, treating each feature as a player contributing to the "payout" of a prediction, and distributes credit for the outcome fairly across features. SHAP explanations are local by default but aggregate cleanly into global feature importance.

A third family answers a more actionable question. A counterfactual explanation describes the smallest change to an input that would have flipped the outcome: not "why was the loan denied," but "what would have gotten it approved." This is often the explanation a person actually wants, because it points at an action rather than a mechanism.

Alongside these, older and simpler global tools remain useful: permutation feature importance (how much does shuffling one feature hurt accuracy?), partial dependence plots (how does the prediction change as one feature varies, averaged over all others?), and surrogate models (fit an interpretable model to mimic the black box and read the surrogate instead). For image and text models built on neural networks, saliency maps highlight which pixels or tokens most influenced a prediction.

What XAI does not give you

An explanation is not a certificate of correctness, and it is not a causal account of the world. SHAP and LIME describe how the model used its inputs to produce an output; they say nothing about whether the model itself is well-specified or whether its inputs are the true causes of the real-world outcome. A model trained on biased data will produce an explanation that faithfully describes its biased reasoning — the explanation makes the bias visible, but it does not remove it.

There is also a real trade-off between how accurately an explanation reflects a model's internal computation and how easy that explanation is for a person to use. NIST's four principles of explainable AI name this directly: an AI system should provide explanation (evidence or reasoning for its output), that explanation should be meaningful to the specific audience receiving it, it should be held to explanation accuracy — a property distinct from the model's own decision accuracy — and the system should operate within knowledge limits, only producing decisions under the conditions it was built and tested for. A dense, fully faithful explanation and a short, accessible one both satisfy part of this list and neither is free.

Where this fits and when it matters

Explainability sits alongside evaluation and security as part of responsible model operation: you use it during development to debug a model that is right for the wrong reasons, during audit to check for unfair treatment of protected groups, and during deployment to satisfy regulatory obligations. The EU AI Act, for example, requires that high-risk AI systems be designed so their operation is "sufficiently transparent to enable deployers to interpret the system's output and use it appropriately" — a legal transparency obligation that XAI techniques are the practical way to meet.

Not every system needs this machinery. A spam filter with a low cost of error, reviewed by no one, and affecting no one's rights does not justify the overhead of computing Shapley values on every prediction. Reach for XAI when a decision affects a person materially, when someone has to justify it after the fact, or when you need to debug a model whose predictions puzzle you — not as a default step in every pipeline.