Module 11 of 11 / 4 lessons / 7 practice exercises / 22 links
Capstone: one client, one POC, end to end
The capstone is where you run the whole FDE loop yourself, in a pair, on one of the assessment use cases, with a mentor playing the client. This guide walks you through the two weeks and gives you a template for every deliverable: the brief, the options paper, the evaluation sheet, the demo script and the hand-over pack.
New to a word? Check Start here: the basics first. How to use this module. Read each lesson top to bottom: the diagram first, then the explanation and the worked example. Use "Find more here" to go deeper with docs, videos and courses. Then do the practice. Tick "I did this" to track your progress in this browser.
You will know what happens on each day of the two capstone weeks, what you hand in and how it is scored.
Why this matters on a client engagement
On a real engagement nobody tells you the order of the work. You have a client, a problem and a deadline. The capstone copies that. The point is not a perfect POC; it is showing that you can move from a vague problem to a working, measured solution that a client owner trusts, in a fixed time, with a partner.
The capstone in ten working days
Kick-off and last real casesDay 1: notes from the client
Process mapDay 2: swimlane the client corrects
One-page briefDay 3: agreed success measure
Options paperDays 4 to 5: one recommendation
Build the main pathDays 6 to 7: runs end to end
Evaluate and fixDay 8: pass rate and failures
Deploy and rehearseDay 9: URL, README, runbook
Panel demoDay 10: scored on the rubric
Read left to right. Each box ends with something handed to the mock client or the panel.
The short version
You and your partner pick one use case from the assessment list. A mentor plays the client: the process owner, sometimes a user, sometimes IT. You get the same messy material a real client would hand over: sample documents, an export from a system, a policy or two.
Across two weeks you run discovery, write a one-page brief, propose options, build the recommended option, evaluate it on real cases, deploy it behind access control and demo it to a panel. Every deliverable has a template in this module.
Week 11: understand and decide
The first week is about the problem, not the code. Most failed POCs fail here: the team builds something clever for a problem the client does not have.
Day 1: kick-off with the mock client. Ask for the last three real cases, not a description of the process.
Day 2: map the process as a swimlane. Mark where time is lost and where errors happen. Send the map to the client for correction.
Day 3: write the one-page brief with a baseline and a success measure. Get the client to agree to it in writing.
Day 4: write the options paper with two or three options and a recommendation. Start the evaluation set from real cases.
Day 5: options review with the mentor. Leave with a decision and a scope you can build in five days.
Week 12: build, measure, show
Split the work so both partners touch every stage. A common pattern is that the domain person leads discovery and the evaluation set while the software person leads the build, and both swap halfway so each can explain the other's part to the panel.
Days 6 and 7: build the smallest version that handles the main path end to end. Use an AI coding assistant, but you own every line.
Day 8: run the evaluation set. Read every failure. Fix the top causes, then run it again.
Day 9: deploy behind access control, write the README and the runbook, rehearse the demo twice.
Day 10: panel. Ten-minute demo, questions, hand-over pack. Scored on the rubric.
How you are scored
The panel uses the rubric from the program overview: discovery, problem framing and value, solution options, build quality, evaluation and safety, deployment and operations, and communication. Each is scored as not yet, practitioner or FDE-ready.
To graduate you need FDE-ready in discovery, build quality, evaluation and safety, and communication, and at least practitioner in the rest. Notice what is missing: there is no score for how advanced the technology is. A simple solution that is measured and trusted beats a complex one that is not.
Rules of the capstone
Use only the data the mock client gives you, or synthetic data you generate and label as synthetic.
Never paste real personal data into a public tool. Treat the sample data as if it were real client data.
Every claim in the brief, the options paper and the demo must point to evidence: an interview note, a document or a number from the evaluation.
If you change scope, write it down and get the client to agree. Silent scope changes count against you.
Worked example
Worked example: a capstone plan for support ticket triage
Here is how one pair might plan the support ticket case. The client is a head of support with 500 anonymized tickets, a help-center export and an SLA policy.
On day 1 they ask the client to walk through three tickets from yesterday: one easy, one escalated, one that breached the SLA. They learn that the real pain is not writing replies; it is tickets sitting unassigned for hours because nobody owns the shared inbox.
That changes the brief. The success measure becomes time to first assignment, with a baseline they compute from ticket timestamps in the export. Reply drafting becomes a second step, only if time allows.
The options paper compares three options: routing rules only, an LLM classifier with a confidence threshold and human review below it, and a full agent that also drafts replies. They recommend the second and explain why the third is too much for five days.
Evaluation set: 60 tickets, labeled with the correct category and priority by the mock client.
Build: a Python script that classifies each ticket with the Claude API, returns JSON, and writes a routing suggestion.
Demo: start with an unassigned ticket from yesterday, show the suggestion, show the pass rate, show the three failure types and what a human does with them.
Start practicing
Practice 1
Draft your own ten-day plan
45 min
Goal: Turn one assessment use case into a day-by-day plan before the capstone starts.
Open the program overview and read all seven assessment use cases.
Pick two that interest you and your partner. For each, write one line on what you think the real pain might be.
Choose one. Copy the ten-day flow above into a document and write, for each day, the exact output you will hand over.
List the questions you will ask the mock client on day 1. Aim for at least eight, all about real past cases.
Ask Claude or another LLM to critique the plan: what is missing, what is too big for ten days. Keep the changes you agree with and note why.
Done when: You have a one-page plan with a named output for every day and a day-1 question list your partner has reviewed.
Common mistakes
Starting to code on day 1. You will build for the problem you imagined, not the one the client has.
Letting one partner do all the building. The panel asks both of you about every part.
Treating the mock client as a formality. They hold information you only get by asking about real cases.
Choosing the most advanced option to impress the panel. The rubric rewards fit, measurement and trust.
Check yourself
What must be true by the end of day 5 before you start building?
Which four rubric dimensions must be FDE-ready to graduate?
Why does every claim in the demo need evidence, and what counts as evidence?
How do you handle a scope change in the middle of the build?
Lesson 11.2
Templates: the one-page brief and the options paper
You will be able to write a brief the client signs off and an options paper that ends in a clear, defensible recommendation.
Why this matters on a client engagement
The brief and the options paper are the two documents that protect you and the client. The brief fixes what success means before anyone builds. The options paper shows you considered more than one way and chose for good reasons. On a real engagement these are what a sponsor reads, forwards to their manager and remembers.
Choosing what to recommend
Brief agreed with a baseline
Can most of the problem be solved without an LLM?
Yes
Recommend the rule, form or report. Mention AI as a later step if a gap remains.
Partly
Recommend rules for the common path and an LLM for the messy cases, with human review.
No
Recommend the simplest LLM design that meets the measure, with a review gate sized to the risk.
Start from the brief. Work down the question before you pick an option.
The one-page brief
The brief fits on one page because a sponsor will read one page. Every field answers a question the sponsor will otherwise ask you later. Fill it with evidence from discovery, not guesses.
Problem: one or two sentences in the client's words. Who suffers, how often, with what effect.
Current process: a link to the process map and the one step where most time or errors happen.
Baseline: the number today, and where it came from. For example, median hours from ticket created to first assignment, computed from the export.
Success measure: the number you will move and the target. One primary measure; at most two secondary ones.
Scope: what is in, and just as important, what is out.
Users and owner: who uses it, who owns the process, who approves the result.
Data and access: what data the solution reads and writes, and any limits on where it may go.
Risks and constraints: privacy, regulation, systems you cannot touch, people who must agree.
Acceptance criteria: three to six checkable statements. The POC is done when all of them pass.
Writing acceptance criteria that can be checked
An acceptance criterion is a statement you can mark true or false without arguing. A weak criterion says the classifier is accurate. A strong one says that on the agreed evaluation set of 60 tickets, the suggested category matches the label for at least an agreed share of tickets, and every ticket below the confidence threshold is sent to a person.
Agree the targets with the client before you build. If you set them after you see the results, nobody will trust them.
The options paper
The options paper is two to three pages. It describes two or three real options, compares them on the same criteria and recommends one. Include a non-AI option when one is plausible; sometimes a rule, a form or a report solves most of the problem, and saying so builds trust.
Context: one paragraph linking back to the brief.
Options: for each, what it is, how it works in the process, what it needs (data, access, people) and a rough cost to build and run.
Comparison: a table with the same rows for every option: expected effect on the success measure, accuracy risk, cost, time to build, security and privacy, effort to adopt.
Recommendation: which option, why, and what would make you change your mind.
Next step: what you will build in the POC and what is left for later.
Getting sign-off without a meeting marathon
Send the brief as a draft the day after discovery, with the open questions marked. Ask the process owner to correct it, not to approve it; people find mistakes faster than they give approval. Then walk through it together for fifteen minutes and agree the success measure, the target and the out-of-scope list out loud.
Record the agreement in writing: a reply to the email or a comment in the shared document is enough for the capstone. Do the same for the options paper. When the build is under pressure later, this is what lets you say no to a new request politely: it is a good idea, it is outside the brief we agreed, let us put it on the list for the pilot.
Keep both documents short and living. If discovery changes during the build, update the brief, mark what changed and why, and send it again. A brief that matches reality is worth more than one that was perfect on day 3.
Using an LLM to help, honestly
Claude or another LLM is useful for turning interview notes into a draft brief, for finding gaps, and for playing a skeptical sponsor who asks hard questions. It is not a source of facts about the client. Every number and every claim must come from your notes, the documents or the data. A good habit is to ask the model to mark each sentence with the note it came from, and delete any sentence it cannot source.
Worked example
Worked example: a filled brief for HR policy questions
This is an illustrative brief for the HR policy assistant use case. The numbers are examples a learner would compute from the mock client's data, not real statistics.
text
Problem: HR partners answer the same leave and travel questions
by email many times a week; answers vary by who replies.
Current process: see map v2, step 3 (search handbook, reply).
Baseline (example): 120 policy emails in the 4-week sample;
median reply time 6 working hours.
Success measure: share of policy questions answered correctly,
with a citation, without an HR partner.
Target (agreed): set with the HR partner before the build.
Scope in: leave, travel, expenses. Out: pay, performance, legal.
Users: employees. Owner: HR business partner.
Data: handbook and two policies (PDF). No employee records.
Risks: wrong answers on leave entitlement; personal questions.
Acceptance criteria:
1. On the 40-question eval set, answers match the expected
answer at or above the agreed share.
2. Every answer cites the policy section it used.
3. Out-of-scope or personal questions get the HR fallback.
4. No answer is given when no source section is found.
Start practicing
Practice 1
Write a brief from interview notes
60 min
Goal: Produce a one-page brief that a sponsor could sign off.
Pick your capstone use case. Write half a page of pretend interview notes based on its description: client, pain and data.
Fill every field of the brief template from those notes. Leave a field blank rather than guessing.
Write three to six acceptance criteria. Read each one and ask: can I mark this true or false without arguing?
Paste the brief into Claude and ask it to act as a skeptical sponsor and list the five questions it would ask.
Answer those questions in the brief or add them to your day-1 question list.
Done when: Your partner can read the brief in two minutes and tell you the success measure, the scope and how the POC will be judged.
Practice 2
Compare three options in one table
45 min
Goal: Practice making a recommendation you can defend.
Use the same use case. Describe three options: one without an LLM, one with an LLM and human review, and one more automated.
Build a table with the six comparison rows from this lesson and fill it for each option.
Write a three-sentence recommendation, including what would make you change your mind.
Swap with another pair and argue for an option they did not choose.
Done when: You can explain in one minute why your recommended option beats the other two for this client.
Common mistakes
A success measure with no baseline. You cannot show improvement against nothing.
Scope that lists only what is in. The out-of-scope list prevents most later arguments.
Options that are really one option with small changes. Make them genuinely different.
Letting the LLM fill gaps in the brief with plausible text. Every line must trace to evidence.
Check yourself
What are the nine fields of the one-page brief, and which one makes the POC checkable?
Turn this into a checkable criterion: the assistant should be accurate.
Why include a non-AI option in the options paper?
What goes in the comparison table for every option?
Lesson 11.3
Template: the evaluation sheet
You will be able to build a capstone evaluation set of 30 or more real cases, run it, and report a pass rate with the main failures explained.
Why this matters on a client engagement
The evaluation sheet is what turns a demo into evidence. Without it, the panel and the client only see the cases you chose to show. With it, they see how the solution behaves on cases you did not pick, where it fails and what you did about it. It is also the first thing a client asks for when a pilot is discussed.
The evaluation loop
Collect real and edge casesRandom sample plus discovery edge cases
Label expected answersChecked by the mock client
Run every caseOne script, same settings
Read every failureGroup by cause
Fix the biggest causePrompt, retrieval, rule or label
Then back to step 1: Collect real and edge cases
Each pass through the loop should move the pass rate or explain why it cannot move.
What goes in the sheet
Keep it in a spreadsheet or a CSV file in your repository. One row per case. The columns are the same for every use case; only the content changes.
case_id: a stable id so you can talk about a case by number.
input: the ticket text, the document file name or the question.
expected: what a correct answer looks like, labeled by the mock client or by you and checked by the client.
type: normal, edge or out_of_scope, so you can see whether failures cluster.
output: what the solution returned on the latest run.
pass: yes or no against the expected value, using a rule you wrote down before the run.
failure_note: for every no, one line on why.
Where the cases come from
Take most cases from the client's real sample, chosen at random, not the easiest ones. Then add edge cases you heard about in discovery: the scanned invoice, the angry customer with two questions, the policy question that is really about pay. Add a few out-of-scope cases to check that the solution says no when it should.
Thirty cases is the capstone minimum. It is enough to find the main failure types, not enough to claim a precise accuracy figure. Say that in the demo.
Running it and reading failures
Run every case through the solution with one script, so a rerun after a fix takes minutes. Then read every failure. Group failures by cause: missing information in the source, a prompt that is unclear, wrong retrieval, a label that was wrong. Fix the biggest group first and rerun.
Sometimes the failure is the label, not the model. When that happens, agree the change with the client and note it in the sheet. Never quietly edit expected values to raise the pass rate.
Pass rules for free-text answers
Exact matching works for categories, priorities and extracted fields such as an invoice number. Drafted replies, summaries and policy answers need a different rule, because two correct answers can use different words.
Write the rule as a short checklist a person could apply: the answer states the correct entitlement, cites the right policy section, does not add facts that are not in the source, and uses the fallback when the question is out of scope. Score a few cases by hand first to see if the checklist is clear.
Once the checklist is stable you can ask an LLM to apply it, which is called LLM-as-judge. Give the judge the question, the source text, the expected answer, the actual answer and the checklist, and ask for a yes or no per item with a one-line reason. Then check a sample of the judge's decisions yourself. If you disagree with it often, fix the checklist or score by hand; a judge you have not checked is just another unmeasured model.
Reporting the result
Report three things: the pass rate overall and by case type, the top two or three failure causes with an example of each, and what a human does when the solution fails. Show the before and after if a fix moved the number. That is the evidence the rubric asks for.
Worked example
Worked example: scoring a run
This short script reads an evaluation CSV with expected and output columns, marks each row and prints the pass rate overall and by case type. It uses only the Python standard library. Save the sample as eval.csv next to the script and run python3 score.py.
Sample eval.csv to try it: a header line case_id,type,expected,output then rows such as 1,normal,billing,billing and 2,normal,login,billing and 3,edge,refund,refund and 4,out_of_scope,decline,decline.
With those four rows the script prints all 3/4 passed (75%) and one FAIL line for case 2.
Exact matching suits categories. For free text, write a clear pass rule or use a judge prompt, and check a sample of judge decisions by hand.
python
import csv
from collections import defaultdict
# eval.csv columns: case_id,type,expected,output
rows = list(csv.DictReader(open("eval.csv", newline="")))
totals = defaultdict(lambda: [0, 0])
for r in rows:
ok = r["output"].strip().lower() == r["expected"].strip().lower()
r["pass"] = "yes" if ok else "no"
for key in ("all", r["type"]):
totals[key][0] += ok
totals[key][1] += 1
for key, (passed, n) in totals.items():
print(f"{key:13} {passed}/{n} passed ({100 * passed / n:.0f}%)")
for r in rows:
if r["pass"] == "no":
print("FAIL", r["case_id"], "expected", r["expected"], "got", r["output"])
Start practicing
Practice 1
Build a 30-case sheet for your use case
90 min
Goal: Have a real evaluation set ready before you start building.
Create a spreadsheet with the seven columns from this lesson.
Add 20 normal cases from the sample data, picked at random.
Add 7 edge cases from your discovery notes and 3 out-of-scope cases.
Write the expected value for each, and write your pass rule at the top of the sheet.
Ask your mock client or partner to check ten of the labels and fix any they disagree with.
Done when: The sheet has at least 30 labeled rows, a written pass rule and labels a second person has checked.
Practice 2
Run the scoring script
20 min
Goal: See how a pass rate and a failure list are produced.
Copy the script above into score.py.
Create eval.csv with the four sample rows, run python3 score.py and confirm you get 3/4 passed.
Export your own sheet as CSV with the same column names, fill the output column by hand for five rows, and run it again.
Write one line on what the failure list tells you.
Done when: The script runs on your own CSV and prints a pass rate by type and the failing case ids.
Common mistakes
Building the set after the solution, from cases you know it handles. Build it first, from a random sample.
Changing expected labels to improve the score. Change a label only when the client agrees it was wrong, and note it.
Reporting a single accuracy number from 30 cases as if it were precise. Say what it can and cannot tell you.
Skipping out-of-scope cases. A solution that answers everything is a risk, not a feature.
Check yourself
What are the seven columns of the evaluation sheet?
Why should most cases be picked at random from the real sample?
What do you do when a failure turns out to be a wrong label?
What three things do you report from an evaluation run?
Lesson 11.4
Templates: the demo script and the hand-over pack
You will be able to run a ten-minute demo that starts from the client's problem and leave behind a hand-over pack someone else can run with.
Why this matters on a client engagement
The demo is where the sponsor decides whether there is a next step. The hand-over pack decides whether that next step can happen without you in the room. FDEs who skip either one leave behind POCs that nobody uses. The panel scores both, and real clients remember both.
What the hand-over pack contains
Next step and decision neededOne paragraph for the sponsor
Owner and support modelSmall RACI: process, code, user support
Known limits and risksWhat it must not be used for yet
Evaluation sheet and resultsPass rate by type, top failures
RunbookDeploy, secrets location, rollback, checks
READMEWhat it does, run it, run the evaluation
Read top to bottom: from what a sponsor reads first to what an engineer needs last.
The ten-minute demo script
Write the script down and rehearse it twice with a timer. Use real cases from the evaluation set, including one that fails.
Minute 0 to 1: the problem in the client's words and the baseline number.
Minute 1 to 2: the process map with the one step you changed highlighted.
Minute 2 to 6: live run on two real cases: one normal, one hard. Show what the user sees and what the human reviewer does.
Minute 6 to 8: the evaluation result: pass rate by case type, the top failure causes and what you fixed.
Minute 8 to 9: limits and risks, said plainly. What it must not be used for yet.
Minute 9 to 10: the proposed next step, its cost and who needs to say yes.
Handling questions
Expect three kinds of questions: will it work on our other cases, what happens when it is wrong, and what will it cost to run. Answer from the evaluation sheet and the options paper. When you do not know, say so and say how you would find out. A calm, honest I do not know yet, here is how we would measure it, earns more trust than a confident guess.
Setting up so the demo does not break
Live demos fail for boring reasons: an expired key, a slow network, a browser tab logged out. Run the full demo on the deployed URL an hour before, with the same laptop and account you will use. Keep the two demo cases ready in a text file so you paste instead of type.
Have a fallback for every live step: a short screen recording of the same run and screenshots of the results. If the live call fails, say so calmly, switch to the recording and carry on. Panels and clients care much more about how you handle a failure than about the failure itself.
Decide in advance who drives the keyboard and who talks, and swap halfway. Agree a hand signal for running long. Finish on time even if you have to skip a part; the next step at the end matters more than one extra feature.
The hand-over pack
The hand-over pack is a short folder or a section of the repository. Someone who was not in the capstone should be able to run, change and support the POC from it.
README: what it does, how to run it locally, how to run the evaluation, in that order.
Runbook: how to deploy, where secrets live (never the secrets themselves), how to roll back, what to check when it breaks.
Evaluation sheet and the latest results.
Owner and support model: who owns the process, who owns the code, who users contact, written as a small RACI.
Known limits and open risks.
Next step: what a pilot would add, the success measure for the pilot and the decision needed.
The panel day
Each partner presents part of the demo and answers questions on any part. The panel may ask you to run a case they pick, to open the code and explain a function, or to show where a secret is stored. Practice those with your partner the day before.
After the panel
The panel gives written feedback against each rubric dimension. Read it with your mentor and pick two things to improve on your first real engagement. As an FDE graduate you first shadow a live engagement, then co-lead one, then lead. Bring your capstone templates with you: most real engagements need exactly the same brief, options paper, evaluation sheet, demo script and hand-over pack, filled with a real client's evidence.
Worked example
Worked example: a runbook skeleton
This skeleton fits most capstone POCs. Fill each heading with commands and facts from your own project; keep it under two pages.
text
RUNBOOK - Ticket triage POC
What it is: suggests category and priority for new tickets;
low-confidence tickets go to the support lead.
Owner: support operations lead. Code owner: <name>.
Run locally: see README, section 'Run'.
Run the eval: python3 eval/run.py then python3 eval/score.py
Deploy: ./setup.sh deploy (needs access to the project)
Secrets: API key stored in the hosting platform's secret
store as ANTHROPIC_API_KEY. Never in the repo.
Rollback: redeploy the previous tagged version.
When it breaks:
1. Check the status page of the model provider.
2. Check the last 20 log lines for errors.
3. If suggestions look wrong, rerun the eval and compare.
4. If unsure, switch to manual triage and tell the owner.
Limits: English tickets only; not for billing disputes.
Start practicing
Practice 1
Script and time your demo
60 min
Goal: Have a ten-minute demo that starts from the problem and ends with a decision.
Copy the six-part script into a document and write what you will say and show in each part.
Choose two real cases from your evaluation set: one normal, one hard. Note their case ids.
Rehearse with your partner using a timer. Cut anything that pushes you past ten minutes.
Ask your partner to play a skeptical sponsor and ask the three common questions. Note any you could not answer.
Fix the gaps and rehearse once more.
Done when: You can deliver the demo in ten minutes or less and answer the three common questions from evidence.
Practice 2
Assemble the hand-over pack
60 min
Goal: Leave behind something another person can run without you.
Create a handover folder in your repository with the six items from this lesson.
Write the runbook using the skeleton above, with your real commands.
Ask someone outside your pair to follow only the README and run the evaluation.
Fix every place where they got stuck.
Done when: A person who was not in your pair runs the POC and the evaluation from the README alone.
Common mistakes
Opening the demo with the architecture. Start with the client's problem and the baseline.
Showing only cases that work. One honest failure, with what a human does about it, builds trust.
Ending without a clear ask. Every demo ends with a next step and who decides.
A README that only you can follow. Test it on someone else before the panel.
Check yourself
What are the six parts of the ten-minute demo, in order?
How do you answer a question you cannot answer yet?
What six items go in the hand-over pack?
Why must the runbook say where secrets live but never contain them?