Evaluate, deploy and operate

The short version

An evaluation set (eval set) is a list of example inputs, each with the output a competent person would expect. You run your AI solution on every input, compare its answers to the expected ones, and count how often it is right. It is the same idea as a unit test in software or a sample check in a quality team: a fixed set of known cases you can rerun at any time.

The value comes from the word real. Cases you invent at your desk tend to be clean and easy. Cases pulled from the client's actual work contain the messy spelling, missing fields and odd exceptions that break systems in production.

Where the cases come from

Start where the work already leaves a trail. Most teams have one or more of these, and you should ask for them in discovery:

• Historical records: last month's tickets, invoices, claims or emails, together with what a person decided.
• Exception and error logs: the cases that were escalated, reworked or complained about. These are your edge cases.
• Shadowing notes: cases you watched a person handle, where you also wrote down why they decided as they did.
• Expert-written cases: a small number of cases a subject matter expert (SME, the person who knows the work best) writes to cover a rule that rarely appears in the history.
• Production traffic, later: once a pilot runs, sample real requests each week and add the interesting ones.

Labeling expected outputs

A label is the correct answer for a case. For a classification task the label is a category such as approve, hold or reject. For a free text task, such as a summary, the label is usually a short list of things the output must contain and must not contain, because there is rarely one correct wording.

Labels must come from the people who own the work, not from you. Sit with the SME for an hour, go through cases together, and record the reason next to each label. When two experts disagree on a case, that is a finding: the process itself is ambiguous, and the client needs to decide a rule before any AI can follow it. Keep a column for who labeled each case so disputes can be traced.

Coverage and size

Coverage means the set contains every kind of case the solution will meet. Write down the case types first, for example normal invoice, price mismatch, partial delivery, duplicate, foreign currency and missing purchase order. Then make sure each type has several cases. A set with many easy cases and no edge cases will give a high score that means very little.

There is no magic size. A useful starting point for a proof of concept is a few dozen cases that cover every type, which you can label in an afternoon with an SME. Grow it as you learn: every bug found in testing or in the pilot becomes a new case, so the same mistake cannot return unnoticed. Keep a small held-back group of cases you never look at while tuning prompts, so you have an honest final check.

Keeping the set healthy

Store the set as a plain file (CSV or JSON) in the same repository as the code, so it is versioned with every change. Remove or mask personal data such as names, account numbers and addresses before the file leaves the client's systems, and agree with the client where the file may live. Review the set every few weeks: business rules change, and an expected output that was right in March can be wrong in June.

Write each input exactly as the solution will receive it: the full email text, or a path to the real PDF, not your summary of it. If the eval input is cleaner than production input, the score will be too.

The short version

Measuring quality has three parts. First, a metric: a rule that turns each output into pass or fail, or a score, and then sums the results. Second, error analysis: reading the failed cases one by one and grouping them by cause. Third, for outputs that are hard to check with code, a judge: a person or an LLM that grades each output against written rules.

The number tells you how good it is. The error analysis tells you why it is not better. You need both.

Choosing a metric

Pick the simplest metric that matches the business question. Common choices:

• Pass rate (accuracy): the share of cases where the output matches the expected label. Good for classification and extraction.
• Pass rate by case type: the same, split by group. A 90 percent overall rate can hide 40 percent on the edge cases that matter most.
• Field-level match: for extraction, check each field (invoice number, amount, date) separately so you see which field breaks.
• Rule checks: for free text, check things code can verify, such as 'mentions the invoice number' or 'is under 100 words'.
• Cost of errors: some mistakes are worse than others. Approving a duplicate invoice costs money; sending a clean one to review only costs time. Count the dangerous kind separately.

Error analysis: reading the failures

Open every failed case and write one short note on what went wrong. After 20 or 30 notes, patterns appear: the model ignores currency, misreads dates in one format, or the label itself was wrong. Group the notes into causes and count them. Fix the biggest cause first, rerun, and repeat.

Error analysis also finds problems that are not the model's fault: bad source data, a missing business rule, or a label the SME now thinks was wrong. These are often the most valuable findings to report to the client, because they improve the process even without AI.

Comparing runs and reporting

A score only means something next to another score. Keep a results file for every run with the date, the prompt version, the model and the pass rate per case type, so you can say 'version 4 fixed the currency cases but broke two partial deliveries'. Change one thing at a time between runs, or you cannot tell which change helped.

LLM output can vary between runs even with the same input. For a small set, run it two or three times and look at whether the same cases fail; a case that passes and fails at random is a sign the instructions are unclear. When you report to the client, give the pass rate, the number of cases behind it, the dangerous errors, and the main failure causes in plain words.

LLM-as-judge and its limits

When the output is free text, such as a summary or a reply draft, code cannot easily say whether it is good. An LLM-as-judge is a second model call that reads the input, the output and a list of rules, and returns pass or fail with a reason. It is fast and cheap enough to run on every case each time you change the prompt.

Judges have limits. They can be inconsistent between runs, they may prefer longer answers, and they can miss facts that were made up if the rules do not ask about them. Treat the judge as a measuring tool that must itself be checked: have the SME grade 20 or 30 outputs by hand, run the judge on the same outputs, and see how often they agree. Only trust the judge on the kinds of cases where it agrees with the human. Keep each rule narrow and yes or no; a vague question such as 'is this good?' gives vague grades.

Here is a simple judge prompt for grading summaries of supplier dispute emails. The words in curly brackets are placeholders your script fills in for each case.

You are grading a summary of a supplier dispute email.

Email:
<email>{email}</email>

Summary to grade:
<summary>{summary}</summary>

Check each rule and answer PASS or FAIL for each:
1. States the invoice number if the email has one.
2. States what the supplier is asking for.
3. Adds no facts that are not in the email.

Give a one-line reason for every FAIL. Then on the last line write
OVERALL: PASS only if all three rules pass, otherwise OVERALL: FAIL.

The short version

A human review gate is a point in the workflow where the AI's output waits until a person approves, edits or rejects it. An audit trail is the record of what happened at each step: the input, what the AI produced, which version of the prompt and model produced it, who reviewed it, what they decided and why.

Together they let a client use AI while keeping a person responsible, and let anyone reconstruct a decision months later.

Where a person should approve

Not every output needs a person. Review costs time, so place gates where the risk is. Ask three questions about each AI step: what is the harm if it is wrong, can the harm be undone, and does a rule or regulation require a named approver?

• Always review: actions that move money, change a customer record, send something to a customer or regulator, or affect a person's rights.
• Review when unsure: let the AI pass clear cases but route low-confidence or rule-flagged cases to a person.
• Sample review: for low-risk output, a person checks a random sample each week to catch drift.
• No review: internal drafts the user reads and edits anyway, such as a meeting summary.

What to log

Log enough to answer 'what did the system know, what did it say, and who decided?' for any single case. That usually means:

• A case or request id that links every step together.
• The input or a pointer to it, and the AI output exactly as shown to the reviewer.
• The model name and version, and the prompt version (keep prompts in Git so a version maps to exact text).
• The reviewer's identity, their decision (approved, edited, rejected), the final output and a short reason.
• Timestamps for each step.

Why it matters in regulated work

Regulated teams must often show that controls work, not just that they exist. An audit trail lets an auditor pick any case and follow it end to end. It also gives you data: the share of AI suggestions that reviewers edit or reject is one of the best quality signals you will get after go-live. A rising edit rate is an early warning that something has changed.

Design the trail with the client's compliance or risk team, not alone. They know how long records must be kept, who may see them, and whether personal data may sit in a log. Make the log append-only where possible, so entries are added but never changed.

Making review easy

A gate only works if reviewers use it properly. Show the AI output next to the source, highlight what the AI relied on, and make approve, edit and reject one click each with a required reason for edits and rejections. If review is slow or confusing, people will click approve on everything, and the gate becomes theatre.

A common pattern for review when unsure is a routing rule. The AI returns its decision plus the evidence it used, and code (not the model) applies the client's rules: any amount above an agreed limit, any new supplier, any mismatch between the invoice and the PO, or any case the model marks as needs_review goes to a person. Clear matches below the limit can pass, with a weekly sample still checked by hand. Because the rule is in code, the client can read it, change it and show it to an auditor.

Agree with the client what a reviewer should do when the AI is wrong. An edit with a reason is better than a silent fix in another system, because only the first one reaches the audit log and the eval set.

The short version

Deploying means putting your software on a computer that is always on and reachable by its users. There are three common shapes. A static page is a set of files (HTML, CSS, JavaScript) served as they are. Serverless means you upload a function and the platform runs it only when a request arrives. A container packs your app and everything it needs into one image that runs the same way on any machine with a container runtime.

Each shape trades control for simplicity. Static is simplest and does least. Containers give the most control and need the most care.

Static pages

A static site has no code running on the server. The host simply sends files. That makes it cheap, fast and hard to break. Hosts such as Cloudflare Pages or GitHub Pages can publish a folder from a Git repository in minutes.

Use it for a demo front end, a report, documentation or a dashboard that reads from an API. Do not use it for anything that needs a secret, such as an LLM API key: everything in a static page is visible to anyone who opens it.

Serverless functions

Serverless platforms such as AWS Lambda or Cloudflare Workers run your code on demand. You do not manage a server; you pay per request or per amount of compute time, and the platform scales up and down for you. This is a good fit for an API that calls an LLM: the function holds the API key safely on the server side, takes a request, calls the model and returns the answer.

Limits to know: each run has a maximum duration and memory, there may be a short delay when a function starts after being idle (a cold start), and long-running jobs or large libraries can be awkward. Each provider documents its own limits.

Containers

A container image is built from a recipe called a Dockerfile. It contains the operating system libraries, the language runtime, your code and its dependencies. Because the image is the same everywhere, 'it works on my machine' becomes 'it works on every machine'. Containers run on services such as AWS ECS, Google Cloud Run, Azure Container Apps, or a Kubernetes cluster the client already runs.

Use containers when you need a long-running process, specific system libraries, a background worker, or when the client's IT team requires everything to run inside their own cloud account. Many enterprise clients prefer containers because their security and platform teams already know how to review and operate them.

Here is a complete Dockerfile for the small Python service in the worked example below. Each line is one instruction: start from an official Python image, set the working folder, copy the code in, set a default port and say which command starts the app.

FROM python:3.12-slim
WORKDIR /app
COPY app.py .
ENV PORT=8000
EXPOSE 8000
CMD ["python", "app.py"]

How to choose

Start from the client, not from your favourite tool. Ask where their other apps run, what their security team approves, and who will maintain it. Then pick the simplest option that meets the need. A common shape for an AI proof of concept is a static front end plus one serverless or container API that holds the secrets and calls the model.

Expect the client's IT and security teams to ask the same questions whatever you choose: which cloud account it runs in and who owns that account, where the data is stored and processed, how users sign in, which outside services receive data (including the LLM provider), how it is backed up, and how it is patched. Write the answers on one page early. It shortens the security review and makes the hand-over easier, because the next owner gets the same page.

The short version

CI (continuous integration) means every change pushed to the shared code repository is automatically built and tested. CD (continuous delivery or deployment) means changes that pass are automatically made ready to release, or released. Environments are separate copies of the system for different purposes. Secrets are values such as API keys and passwords that must never appear in code.

Put together: a developer pushes a change, CI runs the tests and the eval set, and only if they pass does the change move to a test environment and then to production.

Dev, test and prod

Most teams use at least three environments:

• Dev (development): where builders try things. It can break. It uses fake or masked data.
• Test (also called staging or QA): a copy that looks like production, where the client's users check a release before it goes live.
• Prod (production): what real users rely on. Changes arrive only after passing test, and access is limited.

GitHub Actions basics

GitHub Actions is GitHub's built-in CI/CD service. You describe a workflow in a YAML file under .github/workflows/ in your repository. A workflow has triggers (on: when to run, such as on a pull request), jobs (groups of steps that run on a fresh virtual machine, the runner), and steps (each either a shell command with run: or a ready-made action with uses:).

For an AI project the most useful step is an eval gate: a script that runs the eval set and exits with an error if the pass rate drops below an agreed threshold. A failed step marks the pull request red, so a prompt change that breaks quality cannot be merged by accident.

Secrets management

The rule is simple: secrets live outside code and are injected at run time. In code, read them from environment variables. Locally, keep them in a .env file listed in .gitignore. In CI, store them as repository or environment secrets in GitHub and pass them in with ${{ secrets.NAME }}; GitHub masks them in logs. In the cloud, use the provider's secret store, such as AWS Secrets Manager, Azure Key Vault or Google Secret Manager.

Use a different key for each environment, so a leaked dev key cannot touch production, and give each key the smallest permissions it needs. If a key is ever committed to Git, assume it is leaked: revoke it and issue a new one. Deleting the commit is not enough, because the history and any copies still hold it.

The same habit inside the code: read the key from the environment and stop with a clear message if it is missing, without ever printing the value.

import os, sys

def require(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        sys.exit(f"Missing required setting {name}. Set it in the environment, not in code.")
    return value

APP_ENV = os.environ.get("APP_ENV", "dev")
API_KEY = require("ANTHROPIC_API_KEY")
print(f"Running in {APP_ENV}; key loaded ({len(API_KEY)} chars, value not printed)")

Configuration the same way

Settings that differ between environments, such as the model name, a database address or a feature flag, should also come from environment variables, not from edited code. The Twelve-Factor App calls this storing config in the environment. It means the exact same build can move from test to prod with only its settings changed.

Plan the way back as well as the way forward. A rollback is returning to the last version that worked. If every release is a tagged build that was tested, rolling back is just deploying the previous tag. Prompts are part of the release too: keep them in Git with the code, so a prompt change goes through the same pull request, eval gate and review as any other change, and can be rolled back the same way.

The short version

Observability means being able to understand what a running system is doing from the data it records. Three kinds of data matter. Logs are timestamped records of events. Traces link all the steps of one request together, such as retrieve documents, call the model, run a tool, so you can see where time and errors went. Metrics are numbers over time, such as requests per hour, error rate, latency and cost.

Alerts sit on top: rules that notify a person when a metric crosses a limit.

What to trace in an LLM app

For each request, record enough to replay and judge it:

• A trace id shared by every step, and the user or case id.
• The prompt version and model name, so changes can be tied to results.
• The input and the output (or masked versions if they hold personal data).
• Retrieved documents or tool calls and their results, for RAG and agent apps.
• Input and output token counts, latency per step, and any error.
• The outcome: was it accepted, edited or rejected by the user or reviewer, and any feedback score.

Cost per request

LLM providers charge per token, with different prices for input and output tokens. Cost per request is input tokens times the input price plus output tokens times the output price. Multiply by expected volume to get a monthly figure, and compare it with the cost of the manual work it replaces. Prices change, so read the current numbers from the provider's pricing page and keep them in configuration, not in code.

Watch for cost drivers: long system prompts sent on every call, many retrieved documents, agent loops that make several calls per request, and retries. Features such as prompt caching and smaller models for simple steps can cut cost; measure before and after.

Report cost in the client's terms, not only in tokens. 'About one cent per invoice at today's example prices, against several minutes of a clerk's time' is a sentence a sponsor can act on. Track cost per successful outcome too: if one request in five is retried or rejected, the real cost per useful answer is higher than the cost per call.

Monitoring drift

Drift is a slow change in inputs or results over time. New invoice layouts, a new product line or a model update can all lower quality without any error appearing. Watch signals that move when quality moves: the reviewer edit or reject rate, user feedback, the share of cases sent to needs_review, and the pass rate when you rerun the eval set each week on fresh sampled cases. Add the surprising cases to the eval set.

Tools and alerts

You can start with structured logs (one JSON object per line) sent to whatever log system the client already uses. LLM-specific tools add trace views, cost dashboards and eval scoring. Langfuse is one open source example that can be self-hosted or used as a cloud service; OpenTelemetry is an open standard for traces that many tools accept. Pick what the client can run and keep after you leave.

Set a few alerts that a person will act on: error rate above a limit, latency above a limit, daily cost above budget, and edit rate rising week on week. Each alert needs an owner and a first step written in the runbook. Too many alerts get ignored.