The reliability library

Understand the failures. Keep the lessons.

Understand the failures. Keep the lessons.

Understand the failures. Keep the lessons.

Engineering field notes, practical methods and longer reads on making AI work reliably—from the source it reads to the outcome a customer receives.

From the engineering field notes

Small failures. Consequential outcomes.

Small failures. Consequential outcomes.

Small failures. Consequential outcomes.

Document review

Whole-file coverage comes before finding accuracy.

Whole-file coverage comes before finding accuracy.

Whole-file coverage comes before finding accuracy.

A clean queue can hide an obligation that never became a finding.

Read the note ↗

Document review

A dismissed finding is not automatically wrong.

A dismissed finding is not automatically wrong.

A dismissed finding is not automatically wrong.

Actionability and correctness answer different questions.

Read the note ↗

Browse the library

Browse the library

Browse the library

Original publication dates are retained. Earlier articles include historical perspectives on transformation, governance and evaluation.

Article

A flattened judge can miss failures in long agent sessions

A flattened judge can miss failures in long agent sessions

A flattened judge can miss failures in long agent sessions

Why a long agent session needs an investigation of its tool actions, state and evidence—not just a verdict on the final response.

Read the article ↗

Guide

AI evidence worksheet library

AI evidence worksheet library

AI evidence worksheet library

Self-guided questions for recording workflow evidence, owners, gaps and next checks.

Read the article ↗

Article

PE portco control pattern.

PE portco control pattern.

PE portco control pattern.

A resource version of the PE-portco AI audit pattern: visibility, unauthorized MCP paths, sanctioned-tool cadence, and board-ready evidence.

Read the article ↗

Article

Shadow MCP audit methodology for finance.

Shadow MCP audit methodology for finance.

Shadow MCP audit methodology for finance.

An 8-layer operating read for finding unmanaged MCP servers, agent connectors, OAuth grants, and AI delegation risk before the board asks.

Read the article ↗

Diagnostic

Shadow AI and connector inventory worksheet

Shadow AI and connector inventory worksheet

Shadow AI and connector inventory worksheet

Start with the surfaces the team can actually inspect. This worksheet helps record coverage and unanswered questions; it does not scan endpoints, detect connectors or calculate exposure.

Read the article ↗

Article

How financial institutions get mis-sold AI.

How financial institutions get mis-sold AI.

How financial institutions get mis-sold AI.

How finance leaders can separate real AI governance risk from fear-selling by asking for measurement, materiality, traces, thresholds, and operating evidence.

Read the article ↗

Article

AI Agent Gateways: What They Catch, and What They Miss

AI Agent Gateways: What They Catch, and What They Miss

AI Agent Gateways: What They Catch, and What They Miss

A finance operating view on where AI agent gateways help, where they miss release, representation, and training-time failures, and why visibility and evaluation come before control.

Read the article ↗

Article

Agents scale execution. Responsibility still needs a seat.

Agents scale execution. Responsibility still needs a seat.

Agents scale execution. Responsibility still needs a seat.

Why agents can take on execution but cannot take on responsibility, and why finance firms need an independent AI audit seat that can produce board-reliable evidence.

Read the article ↗

Article

AI fluency is a learning curve.

AI fluency is a learning curve.

AI fluency is a learning curve.

What Anthropic's March 2026 Economic Index implies for measuring workforce AI fluency inside finance firms.

Read the article ↗

Guide

AI in Insurance: How to Build Trust.

AI in Insurance: How to Build Trust.

AI in Insurance: How to Build Trust.

How insurers can build trust in AI with correctness evidence, audit trails, human approval gates, and regulator-ready proof for core modernization.

Read the article ↗

Article

What broke when the AI met the workflow.

What broke when the AI met the workflow.

What broke when the AI met the workflow.

Why AI workflows fail when buyer risk tolerance, downstream authority, and measurement are misaligned, plus a two-question diagnostic for finance leaders.

Read the article ↗

Guide

Golden datasets for AI evaluation.

Golden datasets for AI evaluation.

Golden datasets for AI evaluation.

A golden dataset is the labeled benchmark that lets you baseline AI behavior and detect drift over time. Here is what it is, how it differs from a golden record, and the five traps most teams fall into.

Read the article ↗

Article

Golden record vs. golden dataset.

Golden record vs. golden dataset.

Golden record vs. golden dataset.

A golden record is a master-data-management concept. A golden dataset is an AI-evaluation concept. Here is the one-minute version of how to tell them apart.

Read the article ↗

Guide

Solutions for AI compliance in financial institutions.

Solutions for AI compliance in financial institutions.

Solutions for AI compliance in financial institutions.

AI compliance solutions for financial institutions: inventory, regulatory classification, monitoring, audit evidence, and finance-ready AI governance.

Read the article ↗

Article

Why Bank AI Budgets Get Approved and Don't Ship

Why Bank AI Budgets Get Approved and Don't Ship

Why Bank AI Budgets Get Approved and Don't Ship

Why AI budgets in banks get approved but fail to produce business outcomes, and how finance leaders can change the reward function.

Read the article ↗

Guide

NL-to-SQL evals for finance.

NL-to-SQL evals for finance.

NL-to-SQL evals for finance.

A practical guide to evaluating NL-to-SQL systems in finance with answer correctness, dataset quality, golden datasets, drift gates, and audit memoranda.

Read the article ↗

Guide

What are NL-to-SQL evals?

What are NL-to-SQL evals?

What are NL-to-SQL evals?

A practical definition of NL-to-SQL evals for finance teams: what they test, why text-to-SQL benchmarks are not enough, and what evidence production systems need.

Read the article ↗

Template

How to build a golden dataset for NL-to-SQL.

How to build a golden dataset for NL-to-SQL.

How to build a golden dataset for NL-to-SQL.

How finance teams should build a persona-first golden dataset for NL-to-SQL systems, with materiality thresholds, tenant slices, role slices, and expected answer metadata.

Read the article ↗

Guide

Answer correctness and dataset quality are different evals.

Answer correctness and dataset quality are different evals.

Answer correctness and dataset quality are different evals.

Why production NL-to-SQL systems in finance need two eval surfaces: answer correctness on the final response and dataset quality underneath the query.

Read the article ↗

Guide

NL-to-SQL fails differently in finance.

NL-to-SQL fails differently in finance.

NL-to-SQL fails differently in finance.

The main NL-to-SQL failure modes finance teams should evaluate: routing drift, semantic mismatch, data-quality failure, RBAC errors, SQL mistakes, and misleading output rendering.

Read the article ↗

Guide

How to audit an NL-to-SQL system.

How to audit an NL-to-SQL system.

How to audit an NL-to-SQL system.

A step-by-step audit workflow for NL-to-SQL systems in finance, including scope, materiality, golden datasets, trace replay, exceptions, drift gates, and audit memoranda.

Read the article ↗

Guide

Semantic layer evaluation for finance AI.

Semantic layer evaluation for finance AI.

Semantic layer evaluation for finance AI.

How to evaluate the semantic layer behind finance AI and NL-to-SQL systems, including metric meaning, tenant overlays, synonyms, sample values, and role-aware retrieval.

Read the article ↗

Checklist

NL-to-SQL evaluation checklist.

NL-to-SQL evaluation checklist.

NL-to-SQL evaluation checklist.

A practical NL-to-SQL evaluation checklist for finance teams, covering scope, golden datasets, semantic layer checks, answer correctness, dataset quality, drift, CI gates, and audit memoranda.

Read the article ↗

Template

AI audit memorandum template.

AI audit memorandum template.

AI audit memorandum template.

A practical audit memorandum template for AI systems, including opinion, scope, materiality, exceptions, remediation, working papers, and evidence sections.

Read the article ↗

Guide

AI gateways for finance.

AI gateways for finance.

AI gateways for finance.

How finance teams should understand AI gateways: routing, identity, policy, telemetry, model access, and evidence inside the AI operating stack.

Read the article ↗

Guide

AI agent security for finance.

AI agent security for finance.

AI agent security for finance.

A finance operating view of AI agent security: tool permissions, API inventory, prompt injection, data exposure, runtime traces, and governance evidence.

Read the article ↗

Guide

Shadow MCP discovery.

Shadow MCP discovery.

Shadow MCP discovery.

How finance teams should discover unmanaged MCP servers, developer agents, local connectors, and tool permissions as part of a Shadow AI Audit.

Read the article ↗

Guide

What is an AI Audit?

What is an AI Audit?

What is an AI Audit?

An AI Audit maps approved tools, Shadow AI, embedded AI, internal agents, spend waste, risk exposure, and adoption outcomes into one finance-ready operating read.

Read the article ↗

Guide

The AI Audit checklist.

The AI Audit checklist.

The AI Audit checklist.

A practical AI Audit checklist for finance teams: inventory, Shadow AI, usage depth, spend waste, risk exposure, eval coverage, and board-ready evidence.

Read the article ↗

Guide

Shadow AI needs an audit, not a panic.

Shadow AI needs an audit, not a panic.

Shadow AI needs an audit, not a panic.

A Shadow AI Audit finds AI tools, MCP servers, embedded features, and internal agents outside the approved estate, then maps ownership, exposure, and controls.

Read the article ↗

Guide

Enterprise AI Audit, built for finance.

Enterprise AI Audit, built for finance.

Enterprise AI Audit, built for finance.

An enterprise AI Audit gives finance leaders one operating read across vendor tools, embedded SaaS AI, internal agents, Shadow AI, value, risk, and evidence.

Read the article ↗

Guide

AI Audit and AI Governance work together.

AI Audit and AI Governance work together.

AI Audit and AI Governance work together.

AI Audit and AI Governance work together: the Audit produces the operating read, while Governance turns material risk and evidence gaps into controls.

Read the article ↗

Guide

AI Trust for Finance.

AI Trust for Finance.

AI Trust for Finance.

Finance AI trust diagnostic: one operating read across AI value, AI risk, Shadow AI, internal agents, model-risk exposure, spend waste, and board-ready evidence.

Read the article ↗

Framework

The Eval Maturity Model.

The Eval Maturity Model.

The Eval Maturity Model.

An 8-stage eval maturity model for AI teams: from manual spot checks to golden sets, checkpoint comparators, CI gates, feedback capture, and multi-tenant launch gates.

Read the article ↗

Template

Golden Set YAML Template.

Golden Set YAML Template.

Golden Set YAML Template.

A ready-to-adapt golden set YAML template for AI products, with an intent layer, tenant-specific data layer, status workflow, checkpoint order, and known failure modes.

Read the article ↗

Framework

AI governance self-assessment worksheet

AI governance self-assessment worksheet

AI governance self-assessment worksheet

Use the questions to expose evidence gaps in a live workflow. Answer with a source and an owner rather than an unsupported confidence rating.

Read the article ↗

Article

Best AI Solutions for Financial Risk Management in 2026.

Best AI Solutions for Financial Risk Management in 2026.

Best AI Solutions for Financial Risk Management in 2026.

Best AI solutions for financial risk management in 2026: what to ask of any vendor, and how to keep the AI agents in your risk workflows reliable in production.

Read the article ↗

Chapter

The AI operating stack for finance.

The AI operating stack for finance.

The AI operating stack for finance.

A finance-first map of the AI operating stack: AI Audit visibility, gateways, agent security, governance evidence, transformation value, and workforce fluency.

Read the article ↗

Article

Prove adoption works with evaluation.

Prove adoption works with evaluation.

Prove adoption works with evaluation.

Why seats, sessions, and tool counts do not prove AI is working, and why finance AI programs need continuous evaluation before governance becomes defensible.

Read the article ↗

Chapter

Why frameworks tell you what to track, but not where the threshold sits.

Why frameworks tell you what to track, but not where the threshold sits.

Why frameworks tell you what to track, but not where the threshold sits.

Frameworks tell you what to track in AI. They don't tell you what 'good enough' looks like. That gap is the practical heart of AI assurance.

Read the article ↗

Chapter

Adoption -> Assurance. The sequence every AI team walks.

Adoption -> Assurance. The sequence every AI team walks.

Adoption -> Assurance. The sequence every AI team walks.

You can't sell governance to an organization that hasn't solved adoption. The sequencing insight at the center of enterprise AI tooling.

Read the article ↗

Chapter

From tools-deployed to people-skilled.

From tools-deployed to people-skilled.

From tools-deployed to people-skilled.

Workforce fluency is the measurable skill of using AI to do real work · role-specific tooling, hands-on training, prompt libraries, and adoption telemetry.

Read the article ↗

Chapter

Why continuous beats periodic.

Why continuous beats periodic.

Why continuous beats periodic.

Point-in-time attestation was built for deterministic systems. AI isn't deterministic. Why audit-as-a-stream beats audit-as-an-event.

Read the article ↗

Chapter

The 12 Levers of Enterprise AI Adoption.

The 12 Levers of Enterprise AI Adoption.

The 12 Levers of Enterprise AI Adoption.

A finance operating model for where AI creates value, where risk forms, who owns the lever, and what evidence proves progress.

Read the article ↗

Diagnostic

AI reliability maturity worksheet

AI reliability maturity worksheet

AI reliability maturity worksheet

Use maturity as a set of evidence questions about one workflow. A team can be strong in one dimension and weak in another; no single score establishes readiness.

Read the article ↗

Scorecard

AI adoption evidence worksheet

AI adoption evidence worksheet

AI adoption evidence worksheet

Decide which workflow deserves the next investment by examining its current outcome, constraints and evidence. Use this worksheet in a team discussion; it is not a peer ranking.

Read the article ↗

Scorecard

AI strategy evidence worksheet

AI strategy evidence worksheet

AI strategy evidence worksheet

Connect the business thesis to a specific workflow, the people who run it and the evidence that would justify continued investment.

Read the article ↗

Scorecard

AI workflow improvement worksheet

AI workflow improvement worksheet

AI workflow improvement worksheet

Translate a broad transformation ambition into one observable change in how a customer job gets done. Record the baseline before proposing the improvement.

Read the article ↗

Scorecard

AI governance evidence worksheet

AI governance evidence worksheet

AI governance evidence worksheet

Assess whether the team can explain what an agent may do, who owns its decisions and what happens when the work diverges from policy.

Read the article ↗

Framework

AI compliance evidence worksheet

AI compliance evidence worksheet

AI compliance evidence worksheet

Organize the evidence for a framework discussion. This worksheet does not determine legal applicability, confer certification or establish that controls are effective.

Read the article ↗

Framework

Agent assurance evidence worksheet

Agent assurance evidence worksheet

Agent assurance evidence worksheet

Prepare questions for an agent-level assurance review. The worksheet organizes evidence; it does not certify an agent or replace the current standard or a qualified assessor.

Read the article ↗

Article

When the Control Drifts

When the Control Drifts

When the Control Drifts

A joint Tensile and Accorian framework on control drift, runtime detection, autonomy budgets and production evidence.

Read the article ↗

Put the learning to work.

Put the learning to work.

Put the learning to work.

Bring the customer job, the expected result and the evidence your team would accept.

Discuss a concrete workflow ↗