The reliability library
Engineering field notes, practical methods and longer reads on making AI work reliably—from the source it reads to the outcome a customer receives.
From the engineering field notes
Original publication dates are retained. Earlier articles include historical perspectives on transformation, governance and evaluation.
Article
Why a long agent session needs an investigation of its tool actions, state and evidence—not just a verdict on the final response.
Read the article ↗
Guide
Self-guided questions for recording workflow evidence, owners, gaps and next checks.
Read the article ↗
Article
A resource version of the PE-portco AI audit pattern: visibility, unauthorized MCP paths, sanctioned-tool cadence, and board-ready evidence.
Read the article ↗
Article
An 8-layer operating read for finding unmanaged MCP servers, agent connectors, OAuth grants, and AI delegation risk before the board asks.
Read the article ↗
Diagnostic
Start with the surfaces the team can actually inspect. This worksheet helps record coverage and unanswered questions; it does not scan endpoints, detect connectors or calculate exposure.
Read the article ↗
Article
How finance leaders can separate real AI governance risk from fear-selling by asking for measurement, materiality, traces, thresholds, and operating evidence.
Read the article ↗
Article
A finance operating view on where AI agent gateways help, where they miss release, representation, and training-time failures, and why visibility and evaluation come before control.
Read the article ↗
Article
Why agents can take on execution but cannot take on responsibility, and why finance firms need an independent AI audit seat that can produce board-reliable evidence.
Read the article ↗
Article
What Anthropic's March 2026 Economic Index implies for measuring workforce AI fluency inside finance firms.
Read the article ↗
Guide
How insurers can build trust in AI with correctness evidence, audit trails, human approval gates, and regulator-ready proof for core modernization.
Read the article ↗
Article
Why AI workflows fail when buyer risk tolerance, downstream authority, and measurement are misaligned, plus a two-question diagnostic for finance leaders.
Read the article ↗
Guide
A golden dataset is the labeled benchmark that lets you baseline AI behavior and detect drift over time. Here is what it is, how it differs from a golden record, and the five traps most teams fall into.
Read the article ↗
Article
A golden record is a master-data-management concept. A golden dataset is an AI-evaluation concept. Here is the one-minute version of how to tell them apart.
Read the article ↗
Guide
AI compliance solutions for financial institutions: inventory, regulatory classification, monitoring, audit evidence, and finance-ready AI governance.
Read the article ↗
Article
Why AI budgets in banks get approved but fail to produce business outcomes, and how finance leaders can change the reward function.
Read the article ↗
Guide
A practical guide to evaluating NL-to-SQL systems in finance with answer correctness, dataset quality, golden datasets, drift gates, and audit memoranda.
Read the article ↗
Guide
A practical definition of NL-to-SQL evals for finance teams: what they test, why text-to-SQL benchmarks are not enough, and what evidence production systems need.
Read the article ↗
Template
How finance teams should build a persona-first golden dataset for NL-to-SQL systems, with materiality thresholds, tenant slices, role slices, and expected answer metadata.
Read the article ↗
Guide
Why production NL-to-SQL systems in finance need two eval surfaces: answer correctness on the final response and dataset quality underneath the query.
Read the article ↗
Guide
The main NL-to-SQL failure modes finance teams should evaluate: routing drift, semantic mismatch, data-quality failure, RBAC errors, SQL mistakes, and misleading output rendering.
Read the article ↗
Guide
A step-by-step audit workflow for NL-to-SQL systems in finance, including scope, materiality, golden datasets, trace replay, exceptions, drift gates, and audit memoranda.
Read the article ↗
Guide
How to evaluate the semantic layer behind finance AI and NL-to-SQL systems, including metric meaning, tenant overlays, synonyms, sample values, and role-aware retrieval.
Read the article ↗
Checklist
A practical NL-to-SQL evaluation checklist for finance teams, covering scope, golden datasets, semantic layer checks, answer correctness, dataset quality, drift, CI gates, and audit memoranda.
Read the article ↗
Template
A practical audit memorandum template for AI systems, including opinion, scope, materiality, exceptions, remediation, working papers, and evidence sections.
Read the article ↗
Guide
How finance teams should understand AI gateways: routing, identity, policy, telemetry, model access, and evidence inside the AI operating stack.
Read the article ↗
Guide
A finance operating view of AI agent security: tool permissions, API inventory, prompt injection, data exposure, runtime traces, and governance evidence.
Read the article ↗
Guide
How finance teams should discover unmanaged MCP servers, developer agents, local connectors, and tool permissions as part of a Shadow AI Audit.
Read the article ↗
Guide
An AI Audit maps approved tools, Shadow AI, embedded AI, internal agents, spend waste, risk exposure, and adoption outcomes into one finance-ready operating read.
Read the article ↗
Guide
A practical AI Audit checklist for finance teams: inventory, Shadow AI, usage depth, spend waste, risk exposure, eval coverage, and board-ready evidence.
Read the article ↗
Guide
A Shadow AI Audit finds AI tools, MCP servers, embedded features, and internal agents outside the approved estate, then maps ownership, exposure, and controls.
Read the article ↗
Guide
An enterprise AI Audit gives finance leaders one operating read across vendor tools, embedded SaaS AI, internal agents, Shadow AI, value, risk, and evidence.
Read the article ↗
Guide
AI Audit and AI Governance work together: the Audit produces the operating read, while Governance turns material risk and evidence gaps into controls.
Read the article ↗
Guide
Finance AI trust diagnostic: one operating read across AI value, AI risk, Shadow AI, internal agents, model-risk exposure, spend waste, and board-ready evidence.
Read the article ↗
Framework
An 8-stage eval maturity model for AI teams: from manual spot checks to golden sets, checkpoint comparators, CI gates, feedback capture, and multi-tenant launch gates.
Read the article ↗
Template
A ready-to-adapt golden set YAML template for AI products, with an intent layer, tenant-specific data layer, status workflow, checkpoint order, and known failure modes.
Read the article ↗
Framework
Use the questions to expose evidence gaps in a live workflow. Answer with a source and an owner rather than an unsupported confidence rating.
Read the article ↗
Article
Best AI solutions for financial risk management in 2026: what to ask of any vendor, and how to keep the AI agents in your risk workflows reliable in production.
Read the article ↗
Chapter
A finance-first map of the AI operating stack: AI Audit visibility, gateways, agent security, governance evidence, transformation value, and workforce fluency.
Read the article ↗
Article
Why seats, sessions, and tool counts do not prove AI is working, and why finance AI programs need continuous evaluation before governance becomes defensible.
Read the article ↗
Chapter
Frameworks tell you what to track in AI. They don't tell you what 'good enough' looks like. That gap is the practical heart of AI assurance.
Read the article ↗
Chapter
You can't sell governance to an organization that hasn't solved adoption. The sequencing insight at the center of enterprise AI tooling.
Read the article ↗
Chapter
Workforce fluency is the measurable skill of using AI to do real work · role-specific tooling, hands-on training, prompt libraries, and adoption telemetry.
Read the article ↗
Chapter
Point-in-time attestation was built for deterministic systems. AI isn't deterministic. Why audit-as-a-stream beats audit-as-an-event.
Read the article ↗
Chapter
A finance operating model for where AI creates value, where risk forms, who owns the lever, and what evidence proves progress.
Read the article ↗
Diagnostic
Use maturity as a set of evidence questions about one workflow. A team can be strong in one dimension and weak in another; no single score establishes readiness.
Read the article ↗
Scorecard
Decide which workflow deserves the next investment by examining its current outcome, constraints and evidence. Use this worksheet in a team discussion; it is not a peer ranking.
Read the article ↗
Scorecard
Connect the business thesis to a specific workflow, the people who run it and the evidence that would justify continued investment.
Read the article ↗
Scorecard
Translate a broad transformation ambition into one observable change in how a customer job gets done. Record the baseline before proposing the improvement.
Read the article ↗
Scorecard
Assess whether the team can explain what an agent may do, who owns its decisions and what happens when the work diverges from policy.
Read the article ↗
Framework
Organize the evidence for a framework discussion. This worksheet does not determine legal applicability, confer certification or establish that controls are effective.
Read the article ↗
Framework
Prepare questions for an agent-level assurance review. The worksheet organizes evidence; it does not certify an agent or replace the current standard or a qualified assessor.
Read the article ↗
Article
A joint Tensile and Accorian framework on control drift, runtime detection, autonomy budgets and production evidence.
Read the article ↗
Bring the customer job, the expected result and the evidence your team would accept.
Discuss a concrete workflow ↗
AI reliability, from production to customer outcome.