Explainable AI: Making Model Decisions Legible

A model declines a loan, flags a transaction, or recommends a treatment. Someone asks why. "The neural network said so" is not an answer a regulator, a customer or a clinician will accept — and it's not an answer an engineer debugging the model should accept either. Explainable AI (XAI) is the set of techniques for producing a real answer. The catch is that each technique answers a slightly different question, and choosing the wrong one produces explanations that are confident, plausible and misleading.

Why explanations matter

Four distinct needs drive the field, and they want different things:

• Debugging. Engineers need to know whether the model learned the right thing. The classic cautionary tale is the classifier that detected huskies versus wolves by looking at snow in the background.
• Trust and adoption. A doctor or analyst will only act on a prediction they can sanity-check.
• Recourse. A person denied something deserves to know what would have changed the outcome.
• Compliance. GDPR's provisions on automated decision-making, the EU AI Act's transparency duties for high-risk systems, and US fair-lending rules requiring adverse-action reasons all create legal obligations to explain.

Interpretable by design vs explained after the fact

The first choice is whether to use a model that's transparent by construction or a black box with explanations layered on. Intrinsically interpretable models — linear and logistic regression, small decision trees, generalised additive models (GAMs) and their modern variants like explainable boosting machines — let you read the reasoning directly. For tabular data, they frequently match gradient-boosted trees and neural networks in accuracy, and there's a strong argument that high-stakes decisions on structured data should default to them.

Post-hoc explanation methods treat the model as a black box and probe it. They're the only option for deep networks on images, text and audio, and they're widely used on tree ensembles too. They come with a fundamental caveat: an explanation of a black box is itself a model of the model, and it can be wrong.

The main post-hoc techniques

• Feature attribution (SHAP, LIME). Assign each input feature a contribution to a specific prediction. SHAP is grounded in cooperative game theory (Shapley values) and has efficient exact algorithms for tree models; LIME fits a simple local model around the prediction by perturbing inputs. Both answer "which features pushed this prediction, and how much?" They do not tell you what the model would do under a different input, and they can be unstable — small changes to the input or the background dataset can reshuffle the attributions.
• Saliency and attention maps for images and text highlight which pixels or tokens the model attended to. Intuitive, and repeatedly shown to be unreliable: several popular saliency methods produce nearly the same maps for a trained model and a randomly initialised one.
• Counterfactual explanations. "You would have been approved if your income were $4,000 higher." These directly support recourse, are easy to understand, and expose whether the model relies on features a person can't change. The challenge is generating counterfactuals that are realistic rather than merely nearest in feature space.
• Global surrogates and partial dependence. Fit a simple model to mimic the complex one overall, or plot how predictions change as one feature varies. Useful for understanding general behaviour; can hide interactions.
• Concept-based and mechanistic methods. For deep networks, testing whether human concepts ("stripes", "medical jargon") are represented internally, and — in the newest research — reverse-engineering the circuits inside language models. Promising, still mostly research.

Explaining large language models

Asking an LLM to explain its answer produces a fluent rationale that may bear no relation to the computation that produced the answer — models have been shown to give explanations that omit the factor that actually drove their output. Chain-of-thought output is useful for debugging but is not a faithful trace. For LLM-based systems the practical approaches are retrieval with citations (so the answer is grounded in sources the user can check), structured outputs that expose intermediate steps, and evaluation of faithfulness rather than plausibility. Interpretability research on model internals is advancing quickly but isn't yet a deployable compliance tool.

Choosing and validating an explanation

Start from the audience and the decision. A regulator needs consistent, documented reasons; a data scientist needs to find bugs; a customer needs actionable recourse. Then:

• Prefer interpretable models for high-stakes tabular decisions unless the accuracy gap is real and measured.
• Use more than one method and check they agree; disagreement is a signal.
• Test explanation faithfulness — for example, remove the "important" features and confirm the prediction actually changes.
• Evaluate with the intended users. An explanation that increases confidence without increasing accuracy is worse than none.
• Version and log explanations alongside predictions; you'll need them when a decision is challenged.

Explainability is a component of governance, not a substitute for it. A well-explained biased model is still biased — the explanation just makes it easier to prove.