Explainable AI
Explainable AI develops methods to make machine learning model predictions interpretable by humans. It covers attribution techniques, model-agnostic explanations, inherently interpretable models, and the trust, debugging, and compliance needs that drive the requirement to understand why a model produces a given output.
itArtificial intelligence and machine learning | OpenSkills.info
Intro
Explainable AI
A machine learning model can be extremely accurate and still be a mystery. It takes an input, produces a score or a label, and gives you no account of why. Explainable AI, usually shortened to XAI, is the set of concepts and techniques for closing that gap: turning a model's output into something a person can understand and check.
The problem it solves is trust under uncertainty. When a model denies a loan, flags a transaction as fraud, or recommends a treatment, someone has to answer for that decision — a developer debugging a failure, an auditor checking for bias, a regulator verifying compliance, or the person the decision affects. None of them can do that job from a number alone. They need a reason.
Why "accurate" is not "explainable"
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://www.nist.gov/artificial-intelligence/ai-research-explainability
Supports
- AI must be explainable to society to enable understanding, trust, and adoption of AI technologies
- NISTIR 8312 is NIST's primary deliverable on explainable AI, published September 2021
- https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8312.pdf
Supports
- The four principles of explainable AI — Explanation, Meaningful, Explanation Accuracy, Knowledge Limits
- The Explanation principle obligates systems to supply evidence, support, or reasoning for each decision
- The Meaningful principle means explanations are understandable to individual users/audiences
- Explanation accuracy is distinct from decision accuracy — an explanation may or may not accurately describe how the system reached its conclusion, regardless of whether the decision itself was correct
- The Knowledge limits principle means the system only produces decisions under the conditions it was designed and tested for
- Explanation accuracy and meaningfulness trade off — detailed explanations can be less accessible; simple explanations can be less complete
- https://christophm.github.io/interpretable-ml-book/
Supports
- The book's taxonomy of interpretable models, local model-agnostic methods, global model-agnostic methods, and neural network interpretation
- Interpretability defined as the degree to which a human can understand the cause of a decision, and as the ability to correctly and efficiently predict a method's results
- The local vs. global and model-agnostic vs. model-specific taxonomy
- Black box models as opaque models whose decision-making process is hidden from human understanding
- Techniques covered: LIME, SHAP, counterfactual explanations, partial dependence plots, permutation feature importance, surrogate models, saliency maps
- https://christophm.github.io/interpretable-ml-book/counterfactual.html
Supports
- A counterfactual explanation describes the smallest change to feature values that changes a prediction to a predefined output
- Qualities of a good counterfactual — reaching the target outcome, minimizing feature changes, staying similar to the original instance, remaining realistic/plausible
- The Rashomon effect — multiple valid counterfactuals can exist for one prediction
- https://github.com/marcotcr/lime
Supports
- LIME stands for Local Interpretable Model-Agnostic Explanations
- LIME explains predictions by perturbing inputs, weighting by proximity, and fitting a local linear surrogate model
- LIME is model-agnostic and works with text, tabular, and image classifiers
- https://shap.readthedocs.io/en/latest/
Supports
- SHAP stands for SHapley Additive exPlanations
- SHAP is a game-theoretic approach connecting optimal credit allocation with local explanations using Shapley values
- SHAP provides explainers such as TreeExplainer, LinearExplainer, and DeepExplainer, and works across tabular, text, and image data
- https://docs.cloud.google.com/vertex-ai/docs/explainable-ai/overview
Supports
- Vertex Explainable AI offers feature-based explanations (Sampled Shapley, Integrated Gradients, XRAI) and example-based explanations
- Sampled Shapley suits non-differentiable models; Integrated Gradients suits differentiable models; XRAI highlights salient image regions
- Local explanations are prediction-specific; global explanations aggregate attributions across a dataset
- https://artificialintelligenceact.eu/article/13/
Supports
- Article 13(1) requires high-risk AI systems to be designed so their operation is sufficiently transparent to enable deployers to interpret the system's output and use it appropriately