Module 08 of 11 / 6 lessons / 8 practice exercises / 44 links
From demo to dependable: agentic system design and the FDE portfolio
A working POC is only the start. This module shows how to decide where AI belongs in a process, design for everything that can go wrong, talk through an agentic system from a vague ask to production, and roll it out on top of the client's existing systems. It ends with a portfolio project plan and preparation for FDE interviews, so you can prove these skills to others.
New to a word? Check Start here: the basics first. How to use this module. Read each lesson top to bottom: the diagram first, then the explanation and the worked example. Use "Find more here" to go deeper with docs, videos and courses. Then do the practice. Tick "I did this" to track your progress in this browser.
You will be able to walk a real workflow step by step and decide which steps need plain code, which need a model's judgment and which need a person's approval.
Why this matters on a client engagement
Every company can now rent the same few strong models through an API. That means the model itself is not the advantage. The advantage comes from applying it to one company's real process: knowing exactly which step is slow, which step needs judgment, and which step must stay with a person. Clients often ask for 'AI everywhere'. Your job is to put it only where it earns its place, because every extra model call adds cost, delay and a new way to be wrong.
Supplier document intake: who does what
Code
Model
Person
1
2
3
4
5
6
7
8
9
10
1. Pull new emails from shared inbox
2. Match sender to supplier list
3. Save attachments, convert to text
4. Classify document type
5. Extract fields to a fixed schema
6. Validate fields and match the PO
7. Apply written exception rules
8. Draft query email for a mismatch
9. Approve mismatch or send draft
10. Post to ERP and log every step
Code: 1. Pull new emails from shared inbox
Code: 2. Match sender to supplier list
Code: 3. Save attachments, convert to text
Model: 4. Classify document type
Model: 5. Extract fields to a fixed schema
Code: 6. Validate fields and match the PO
Code: 7. Apply written exception rules
Model: 8. Draft query email for a mismatch
Person: 9. Approve mismatch or send draft
Code: 10. Post to ERP and log every step
Read left to right in time order; scroll sideways on small screens. Only three of the ten steps sit in the model lane.
The short version
Any workflow is a chain of steps. For each step ask one question: what kind of work is this? If the step follows a fixed rule that you can write down (look up a supplier id, check that a total matches a purchase order, send a file to a folder), it is deterministic code. If the step needs reading or writing messy human language (work out what kind of document this is, pull the invoice number out of a scanned PDF, draft a polite reply), it is a job for a model. If the step carries real risk or cannot be undone (pay money, reject a claim, change a customer record), a person approves it.
In practice, when you classify a ten-step process honestly, often only a few steps need a model. The rest is ordinary software that has been missing for years: an inbox rule, an API call, a lookup table. That is good news. Code is cheap, fast, testable and gives the same answer every time.
Why 'send everything to the model' fails
A tempting design is one big prompt: forward every email to the model, attach every file, and ask it to figure out what to do. It demos well on five friendly examples. On real volume it fails in three ways. Cost grows with every token you send, including steps that a single line of code could do for free. Errors grow because each model call has some chance of being wrong, and chaining many calls multiplies those chances. Debugging becomes almost impossible because you cannot tell which part of the long prompt caused the bad decision.
Splitting the work keeps each model call small and focused. A small, focused call is easier to test with an evaluation set (module 9), easier to swap for a cheaper model later, and easier to explain to the client's risk team.
How to classify a step
Walk the workflow with the person who does it today, as you learned in discovery. For each step write down the input, the output, who does it and how long it takes. Then apply these tests in order.
Can I write the rule as if/else, a lookup or an API call? Then it is code, even if a person does it by hand today.
Does the input arrive as free text, images or mixed formats that a rule cannot parse reliably? Then a model reads it and returns structured fields.
Is the output a judgment (category, priority, likely reason) that a person would make by reading? Then a model can propose it, with a confidence value.
Would a wrong answer cost money, break a law or upset a customer, and is it hard to undo? Then a person approves it, at least until the evidence says otherwise.
Does the knowledge live only in one person's head? Then write it down as rules first. Many 'exceptions' turn out to be five or six rules nobody had written.
Hidden rules and the expert in the room
In most messy processes there is one experienced person who knows that supplier X always sends credit notes as screenshots, or that orders from one region need an extra check. These exceptions are not written anywhere. Before you hand them to a model, try to capture them as explicit rules in a small table the client can edit. A rule in a table is visible, testable and cheap. A rule buried in a prompt is invisible to the client and can silently stop working when the model changes.
Keep a short list of exceptions you could not turn into rules. Those are the real candidates for model judgment, and they also become test cases for your evaluation set.
Worked example
Worked example: classifying a 10-step intake process
A distribution company receives supplier documents by email from more than 40 senders. They arrive as PDFs, phone screenshots, spreadsheets and forwarded threads. One accounts clerk knows which suppliers are 'difficult'. Here is the process as she describes it, with a decision for each step.
The result is three model steps, six code steps and one human gate. The model only touches the steps where text is messy or a draft is needed. Everything the model produces passes through code checks before a person or the ERP sees it. When you present this to the client, the person who owns the process can see exactly where AI is used and why.
Step 1, read the shared inbox: code. Use the mail system's API on a schedule.
Step 2, identify the supplier: code. Match the sender address and domain against the supplier master list. Unknown senders go to a person.
Step 3, save attachments and get text: code. Use a PDF text library or an OCR service; this is a tool call, not judgment.
Step 4, decide if this is an invoice, credit note, statement or something else: model. Formats vary too much for rules.
Step 5, extract supplier, invoice number, date, lines and total into a fixed JSON schema: model, followed by validation.
Step 6, check totals add up and match the purchase order: code. Arithmetic and lookups should never be left to a model.
Step 7, apply known exceptions from the clerk (for example, supplier X always rounds totals): code, from a rules table she can edit.
Step 8, write a query to the supplier explaining a mismatch: model drafts it.
Step 9, approve the mismatch handling and send the draft: person. It goes to a supplier and may affect payment.
Step 10, post the matched invoice to the ERP and write an audit record: code.
Start practicing
Practice 1
Classify a workflow you know
40 min
Goal: Practice deciding code, model or human for every step of a real process.
Pick a process you know from work or daily life, such as expense claims, job applications or customer refund requests.
Write it as 8 to 12 numbered steps in a spreadsheet with columns: step, input, output, who does it today.
Add a column 'type' and mark each step code, model or human using the tests in this lesson.
Add a column 'why' with one sentence per decision, and a column 'hidden rules' for any exceptions only an expert knows.
Count how many steps you marked model. If it is more than half, challenge each one: could a rule or lookup do it?
Done when: You have a sheet where every step has a type and a one-sentence reason, and fewer than half of the steps are marked model.
Common mistakes
Starting from the model instead of the workflow. Map the process first, then decide where a model fits.
Letting the model do arithmetic, date math or lookups. These are code and should be exact.
Hiding business rules inside a long prompt where the client cannot see or change them. Keep rules in a table.
Marking every step 'human approval' out of caution. Place gates where risk is real, or people will approve without reading.
Forgetting unknown cases. Every classification step needs a path for 'none of the above'.
Check yourself
What three types of work can a workflow step be, and what test puts a step in each?
Why does one big prompt that 'does everything' become expensive and hard to debug?
Give two examples of steps that look like AI work but should be plain code.
What do you do with exceptions that live only in one person's head?
Lesson 8.2
Designing for the unhappy path
You will be able to design an AI step so that bad output, failed calls and odd inputs are caught, retried or sent to a person instead of breaking the system.
Why this matters on a client engagement
There is one way for a task to go right and many ways for it to go wrong. A demo shows the happy path. A client lives with the unhappy paths every day: the scanned page that is upside down, the supplier who changed their invoice layout, the API that times out at month end. Most of the value of a production system, and most of the trust you build, comes from handling those cases calmly. Module 7 showed how to ask for structured output; this lesson is about what you do when it still goes wrong.
From model output to a safe decision
Model returns outputRetry transient errors with backoff
Parse JSONNot JSON: retry once, then a person
Check schemaMissing fields or wrong types
Check business rulesAllowed values, totals, cross-checks
Check confidenceBelow threshold goes to review
Idempotent write or reviewStable key prevents double writes
Each check can send the case to a person; only output that passes every check runs automatically.
The short version
Treat every model output as untrusted input. Ask for JSON in a defined schema, not free text. Validate it in code, both its shape and its business rules. If a call fails for a temporary reason, retry it with a growing wait. If it keeps failing or the output is invalid or low confidence, route the case to a person with a clear reason. Make every write idempotent, which means doing it twice has the same effect as doing it once, so a retry never creates a duplicate payment or ticket.
None of this is special to AI. It is the same defensive design used for any unreliable network service. The difference is that a model can fail in a new way: it can return something that looks correct but is not.
Build a failure modes catalogue
Before writing more code, list how each step can fail. Use a simple table: failure, how you detect it, what the system does. Fill it from your discovery notes and from the first test runs. Typical entries are below. The catalogue becomes your test plan and, later, the cases you add to the evaluation set in module 9.
Transient errors: timeouts, rate limits, server errors from the model or another API. Detect by error type or status code. Retry with backoff.
Invalid output: not JSON, missing fields, wrong types, extra text. Detect by validation. Retry once with the error message, then route to a person.
Valid but wrong: correct shape, wrong value, for example a total that does not match the line items. Detect with business rules and cross-checks against source data.
Low confidence or unknown case: the document fits no known type. Detect by a confidence field or an 'other' category. Route to a person.
Bad input: empty file, unreadable scan, wrong language, a password-protected PDF. Detect before calling the model and skip the call.
Downstream failure: the ERP or CRM rejects the write. Detect by its response. Do not lose the item; queue it and alert.
Validate shape, then meaning
Shape validation checks that the output parses and has the right fields and types. Pydantic does this well: you define a class with typed fields and call model_validate_json, and it raises a clear error when something is off. For a small script, a hand-written check like the one in the example is fine and has no dependencies.
Meaning validation checks business rules: totals are not negative, the category is in the allowed list, the date is not in the future, the invoice number appears in the source text. These rules catch the 'looks correct but is not' failures that a schema cannot. Keep the rules in code, give each a clear message, and return all problems at once so the reviewer sees the full picture.
Retries, fallbacks and idempotency
Retry only errors that may succeed next time, such as timeouts and rate limits. Do not retry a validation error forever; the same input will usually fail the same way. Use exponential backoff: wait about 1 second, then 2, then 4, plus a small random amount called jitter so many workers do not retry at the same moment. Stop after a fixed number of tries. Official SDKs often retry some errors for you, so check before adding your own loop on top.
A fallback is what you do when the main path fails: try a second model, use a simpler rule-based result, or hand the case to a person. The safest fallback in most client work is a person with all the context attached.
Idempotency matters because retries repeat work. Give each item a stable key, such as the email message id plus attachment name, and check it before writing. Many APIs accept an idempotency key header for exactly this reason. Without it, a timeout after a successful write can make your retry post the same invoice twice.
Routing to a person well
Sending a case to a person is not a failure; it is the design working. Make it useful: put the case in a queue the team already uses, include the original input, the model output, and the exact list of problems found. Track how many cases go to people and why. If one reason dominates, that is your next improvement. Over time the share of automatic cases should grow because you fixed causes, not because you lowered the bar.
Worked example
Worked example: validate and route model output
This script uses only the Python standard library. It takes five outputs like a model might return for supplier documents and decides for each one whether it can go through automatically or needs a person. It runs as written with python3.
Running it prints 'auto' for the first output and a 'human review' line with a reason for each of the other four: a total sent as text, an unknown document type, low confidence, and a reply with chatty text around the JSON. In a real system, the reason string goes into the review queue so the person knows what to look at. With Pydantic you would replace check() with a model class and its validators; the routing idea stays the same.
python
import json
ALLOWED = {"invoice", "credit_note", "statement", "other"}
REQUIRED = {"doc_type": str, "supplier": str, "total": (int, float), "confidence": (int, float)}
def check(raw: str) -> list[str]:
try:
data = json.loads(raw)
except json.JSONDecodeError as err:
return [f"not valid JSON: {err.msg}"]
if not isinstance(data, dict):
return ["output is not a JSON object"]
problems = [f"missing or wrong type: {k}" for k, t in REQUIRED.items()
if not isinstance(data.get(k), t)]
if not problems:
if data["doc_type"] not in ALLOWED:
problems.append(f"unknown doc_type: {data['doc_type']}")
if data["total"] < 0:
problems.append("total is negative")
if data["confidence"] < 0.8:
problems.append("low confidence")
return problems
def route(raw: str) -> str:
problems = check(raw)
return "auto" if not problems else "human review: " + "; ".join(problems)
outputs = [
'{"doc_type": "invoice", "supplier": "Acme Ltd", "total": 1240.5, "confidence": 0.93}',
'{"doc_type": "invoice", "supplier": "Acme Ltd", "total": "1,240.50", "confidence": 0.9}',
'{"doc_type": "receipt", "supplier": "Beta Co", "total": 80, "confidence": 0.95}',
'{"doc_type": "statement", "supplier": "Gamma", "total": 0, "confidence": 0.55}',
'Sure! Here is the JSON you asked for: {"doc_type": "invoice"}',
]
for raw in outputs:
print(route(raw))
Start practicing
Practice 1
Write a failure catalogue and test it
60 min
Goal: Turn 'what could go wrong' into checks that actually run.
Copy the example script into a file and run it with python3 to see the five routing decisions.
In a spreadsheet, list at least eight ways a document intake step could fail, with columns: failure, how to detect, action.
Add two new rules to check() from your list, for example the supplier must not be empty and the total must be under a set limit.
Add three new sample outputs to the list that trigger your new rules, and run the script again.
Ask a free LLM chat to return JSON for a messy invoice text you write, paste its reply into the list, and see how it is routed.
Done when: Your script routes at least eight sample outputs, each new rule fires at least once, and every row in your catalogue has an action.
Common mistakes
Trusting output because it parsed. A valid schema can still hold wrong values; add business rules.
Retrying everything, including validation errors, until you hit rate limits and cost spikes.
Retrying writes without an idempotency key, which creates duplicates after a timeout.
Sending cases to a person with no reason attached, so reviewers redo the whole task.
Hiding failures in logs nobody reads. Count them by reason and review the counts every week.
Check yourself
Name four types of failure in an AI step and how you would detect each.
Which errors should you retry, and why add jitter to backoff?
What does idempotent mean, and how would you make an invoice posting idempotent?
What should a person receive when a case is routed to them?
Lesson 8.3
Agentic system design: from a vague ask to production
You will be able to take a vague request like 'use AI in our factory', ask the right questions, and talk through an MVP and a production design for an agent in a clear order.
Why this matters on a client engagement
Clients rarely arrive with a specification. They arrive with a wish: 'cut emergency room waiting times with AI' or 'use AI in our factory'. The same pattern shows up in the agentic system design round of FDE interviews. In both cases you are judged less on the final diagram and more on how you get there: the questions you ask, the bottleneck you find, the trade-offs you name, and how you keep the system safe when parts of it fail.
From vague ask to production design
Vague ask'Use AI in our factory'
Clarifying questionsUsers, current process, data, rules
Real bottleneckOne sentence, agreed with the sponsor
Requirements and metricOne success measure, what must not fail
Simple MVPSmallest proof on real data, human in control
Production designModel, loop, tools, context, evals, traces
Risks and degradationSecurity, cost, fallback to manual
Move left to right and say each step aloud; do not jump to the design before the bottleneck is clear.
The short version
Work in this order and think out loud. Clarify the ask. Find the real bottleneck. Agree requirements and one success measure. Propose the smallest useful MVP. Only then describe the production design: model, agent loop, tools and permissions, context and memory, evaluation, observability, cost, security, and deployment. Saying your reasoning aloud lets the client or interviewer correct you early, which is cheaper than being wrong late.
Step 1 to 3: clarify, find the bottleneck, define success
Start with questions, not boxes. Who is the user? What happens today, step by step? Where does time or money actually go? What data exists and who owns it? What are the rules (privacy law, safety, union agreements)? What must never go wrong? What would make the sponsor say this worked?
Take the hospital example. 'Waiting time' could mean triage, waiting for a doctor, waiting for test results, or waiting for a bed upstairs. Each has a different cause. If questions reveal that patients wait mostly for a bed because discharge paperwork is slow, the AI opportunity may be drafting discharge summaries for a doctor to approve, not a chatbot in the waiting room. Write the bottleneck as one sentence and one success measure, for example 'median time from discharge decision to bed released, measured weekly'.
Step 4: a simple MVP
The MVP is the smallest thing that proves value on real data with a person in control. For the hospital: a tool that drafts discharge summaries from existing notes for one ward, which a doctor edits and signs. No autonomous actions, no new systems, one integration to read notes. State what is out of scope. A small MVP that works beats a large plan that never ships, and it gives you the evidence to fund the next step.
Step 5: the production design
Walk through the parts below. You do not need to be exhaustive on each; you need to show you have thought about each and know where the risks sit.
Model choice: start with one provider's strongest suitable model to prove quality, then move cheaper models onto simple subtasks such as classification once your evals show they hold up.
Harness and agent loop: the code that calls the model, runs the tools it asks for, feeds results back, and stops on a clear condition or step limit. Prefer a fixed workflow when the steps are known; use an agent loop only when the path truly varies (module 7).
Tools and permissions: each tool does one thing with clear inputs. Read tools are open; write tools are narrow, logged and often need approval. The agent never gets more access than the user it acts for.
Context and memory: what goes into each call (task, relevant documents, recent steps) and what is stored between runs (case history, user preferences). Keep context small and relevant; more is not better.
Evals: a golden set of real cases with expected results, run on every change (module 9).
Observability: a trace per run showing each model call, tool call, input, output, cost and time, so you can answer 'why did it do that?'.
Cost: estimate tokens per case times volume, then cut with caching, smaller models and fewer calls.
Security: prompt injection from documents or emails, jailbreak attempts, personal data masking, least privilege for tools, and a plan for model failures: what the system does when the model is down, slow or wrong. Usually it falls back to the manual process, never to an unsafe action.
Deployment: where it runs, how it is updated, who is on call, and how it rolls out (next lesson).
Worked example
Worked example: answer outline for 'use AI in our factory'
Here is how a strong answer might run, in about the order you would say it. It is an outline, not a script; the point is the sequence and the reasons.
Notice what the answer does not do: it does not start with a model name or a vector database. It earns the design by first finding a narrow, measurable problem.
Clarify: 'What does the factory make, what slows it down today, and who would use the tool? Is there a recent problem that triggered this request?' Suppose the answer is: unplanned machine stops, and technicians spend a long time finding the cause in manuals and old maintenance tickets.
Bottleneck: 'Time from a machine fault to the right fix is long because knowledge is spread across manuals, tickets and a few senior technicians.'
Success measure: 'Median time to repair for the top 20 fault codes on one line, compared with the last three months.'
MVP: 'A fault assistant for one production line. A technician enters the fault code and symptoms; the system searches manuals and past tickets and suggests likely causes with sources. The technician decides. Nothing is written to machines.'
Production design: one model family to start; a fixed workflow of retrieve then answer, with an agent loop only for follow-up questions; read-only tools for manuals, ticket history and sensor readings; context limited to the matched fault code; memory as the ticket history itself, updated by the technician's confirmed fix.
Evals and traces: 100 past faults with known fixes as the golden set; pass rate on 'right cause in top three'; a trace for every query.
Security and failure: no write access to equipment; mask employee names in tickets; if the model is down, the technician uses the manual as today; injection risk is low because sources are internal, but documents are still treated as data, not instructions.
Cost and rollout: estimate queries per shift, start with shadow use by two senior technicians, then widen.
Start practicing
Practice 1
Run a mock design round with a timer
45 min
Goal: Practice the full sequence aloud under time pressure.
Pick one prompt: 'A hospital wants AI to cut emergency room waiting times' or 'A bank wants AI to speed up small business loan decisions'.
Set a 30-minute timer. Spend the first 8 minutes writing clarifying questions and the answers you would assume.
Write the bottleneck in one sentence and one success measure.
Sketch the MVP and the production design on paper using the checklist headings in this lesson.
Record yourself explaining it in 5 minutes on your phone, then play it back and note any checklist item you skipped.
Paste your outline into a free LLM chat and ask it to play a skeptical interviewer with three hard follow-up questions; answer them.
Done when: You have a one-page outline covering every checklist heading, a recording under 6 minutes, and written answers to three follow-up questions.
Common mistakes
Jumping to architecture before asking questions. Spend the first part of the time on the problem.
Designing a fully autonomous agent for an MVP. Start with a person in control and earn autonomy.
Ignoring how the system degrades. Always say what happens when the model is down, slow or wrong.
Giving agents broad write access 'for flexibility'. Narrow tools and least privilege are part of the design.
Thinking silently. In both client meetings and interviews, unspoken reasoning cannot be credited or corrected.
Check yourself
What are the first questions you ask when given a vague AI request, and why?
Why find the bottleneck before choosing a model?
List the headings of a production agent design from model choice to deployment.
How should an agent system behave when the model fails or returns something unsafe?
When would you choose a fixed workflow over an agent loop?
Lesson 8.4
Integrate with what exists and roll out safely
You will be able to plan how an AI solution plugs into a client's current systems and how it moves from test to production in small, low-risk steps.
Why this matters on a client engagement
The person who sponsors your project is taking a personal risk. If the new tool breaks month-end close or sends a wrong email to a key customer, it is their name on it. Clients rarely want to migrate off the ERP, CRM or ticketing system their teams already know. An FDE who builds on top of those systems and rolls out step by step, with evidence at each step, gets adopted. One who asks for a big switch-over often does not.
The rollout ladder
Test environmentSample data, golden set passes
Shadow modeRuns on live cases, writes nothing
Assisted modePerson approves, edits or rejects
Autonomy for low riskNarrow band, widened with evidence
ProductionOwner, monitoring, switch back to manual
Each rung needs evidence from the one before; you can always step back down.
The short version
Leave the system of record where it is. Read from it and write back to it through its official API or import, so people keep working in the screens they know. Then roll out in rungs: a test environment, shadow mode, assisted mode with approval, more autonomy for low-risk cases, and finally production. Keep an audit trail and agent traces from day one so anyone can see what the agent did and why. Report results in business terms: cost saved, risk reduced, revenue gained.
Build on top, not instead
Most companies run on a few core systems: an ERP for finance and orders, a CRM for customers, a ticketing tool for support or IT. They hold the official data, the permissions and the audit history. Your AI component should sit beside them. For example, the invoice assistant reads new invoices from the ERP's inbox queue and writes a suggested match back as a draft that a clerk confirms in the ERP screen.
This approach is close to what Martin Fowler calls the strangler fig pattern: you add new capability around the old system one piece at a time, instead of replacing it in one go. Each piece can be switched off without breaking the rest. Use feature toggles (on/off settings in configuration) so you can turn the AI step off per team or per supplier in minutes if something goes wrong.
The rollout ladder
Each rung has an entry condition, usually a pass rate on your evaluation set or on real cases, and an exit condition agreed with the sponsor. Move up only with evidence, and be ready to move down.
Test environment: run on copies or samples of real data. Nothing reaches real users. Goal: the system works end to end and passes the golden set.
Shadow mode: the AI runs on live cases alongside the people doing the work, but writes nothing. You compare its output with what people actually decided. This is sometimes called a dark launch. Goal: measured agreement on real traffic.
Assisted mode: the AI's output appears as a suggestion or draft, and a person approves, edits or rejects it. Every decision is logged. Goal: high approval rate with few edits.
Increasing autonomy: for clearly low-risk cases (for example invoices under a set amount from known suppliers that match the PO exactly) the system acts on its own, while everything else stays assisted. Widen the low-risk band only as evidence grows.
Production: the agreed scope runs as normal operations, with monitoring, alerts, an owner and a way to switch back to manual.
Audit trail and traces
An audit trail records business decisions: which case, what the AI suggested, who approved it, what was finally done, and when. A trace records the technical path of one run: each model call with its prompt and output, each tool call, timings and cost. Tools such as Langfuse or the tracing built into agent SDKs capture traces for you. Together they let you answer the two questions every client will ask after an incident: what did the agent do, and why. Module 9 covers the audit table and monitoring in more detail.
Measure in the three business buckets
Sponsors fund outcomes, not models. Put every result in one of three buckets. Cost saved: hours of manual work removed, overtime avoided, fewer contractor hours. Risk reduced: fewer errors reaching customers or the ledger, faster detection of fraud or compliance issues, a complete audit trail. Revenue gained: faster quotes, fewer lost orders, quicker onboarding of new customers. Measure a baseline before you start, use the same measure after, and state your assumptions in plain words.
Many engagements start with a paid discovery or audit phase before any build. The output is an operating map of the client's workflows and a value versus effort priority matrix: each candidate use case placed by how much it is worth and how hard it is. The top-right quick wins become the first rollout.
Worked example
Worked example: rolling out an invoice assistant
A mid-sized company wants to speed up invoice matching. The finance director is the sponsor. Their ERP stays the system of record. Here is a rollout plan written in the language the sponsor will read. The numbers are an example of how you would state targets, not results from a real project.
Every stage has an exit decision written down before it starts. That turns 'do we trust the AI?' from a feeling into a check against agreed evidence, which is what lets the sponsor say yes.
Discovery: map the invoice process with two clerks, measure the baseline (for example, average minutes per invoice and the share of invoices with errors found later).
Test: run the assistant on last quarter's invoices in a copy of the data. Target agreed with the sponsor, for example 90 percent correct matches on the golden set.
Shadow, 2 weeks: on live invoices the assistant produces a match; clerks work as usual. Compare daily. Review every disagreement and add the interesting ones to the golden set.
Assisted, 4 weeks: the suggested match appears as a draft in the ERP. Clerks approve or edit. Track approval rate, edit reasons and minutes per invoice.
Low-risk autonomy: exact three-way matches under a set amount from known suppliers post automatically, with a daily sample checked by a clerk.
Report: cost saved (clerk hours), risk reduced (errors caught before payment), revenue effect (early payment discounts captured), each against the baseline.
Start practicing
Practice 1
Write a rollout plan for a sponsor
45 min
Goal: Turn a technical build into a staged plan a business sponsor can approve.
Take the workflow you classified in lesson 1 of this module, or the invoice example.
Name the client's existing systems it must read from and write to, and how (API, file import, shared inbox).
Write one paragraph per rung of the ladder with entry evidence, what users see, and the exit decision.
List what goes in the audit trail and what goes in a trace.
Write one measure for each business bucket (cost, risk, revenue) and how you would get its baseline.
Read the plan to someone without a technical background and note any word they did not understand; replace it.
Done when: You have a one-page plan with five rungs, three business measures with baselines, and no unexplained technical words.
Common mistakes
Asking the client to move data into a new system before you have shown value. Integrate first.
Skipping shadow mode because the demo looked good. Real traffic always contains cases your test set missed.
Moving up the ladder on a feeling. Agree the evidence for each step before you start it.
No way to switch the AI off quickly. Use a feature toggle and keep the manual path working.
Reporting model accuracy to a sponsor who asked about money and risk. Translate into the three buckets.
Check yourself
Why build on top of the client's ERP or CRM rather than replacing it?
What is shadow mode and what do you learn from it?
What is the difference between an audit trail and a trace?
Name the three business buckets and give one measure for each.
What does a discovery or audit phase usually produce?
Lesson 8.5
Example portfolio projects and a 30-day build plan
You will be able to choose, build and write up one end-to-end portfolio project that shows real FDE skills, following a four-week plan.
Why this matters on a client engagement
A generic chatbot or a to-do app shows that you can follow a tutorial. It does not show that you can take a messy business problem, build something that survives real data, prove it works and explain it to a sponsor. That is what hiring teams and clients look for in an FDE. One deep project that does all of this is worth more than many shallow ones.
The 30-day build plan
Pick industry and bottleneckPublic data plus synthetic data
Week 1: one real loopAgent, tools, guardrails, memory, audit
Week 2: make it recoverSchemas, validation, exception handling
Week 4: defend itDeploy, rehearse, get real feedback
Then back to step 1: Pick industry and bottleneck
Each week adds one quality; feedback in week 4 starts the next round of improvements.
The recipe for one strong project
Follow these steps. Each one maps to a skill from this program, so the finished project is evidence of the whole job, not just coding.
Pick an industry you care about or know: healthcare, finance, legal, logistics, HR, manufacturing. Interest keeps you going when the data gets ugly.
Find messy public data, then add synthetic data where real data is private. The Python library Faker generates realistic names, addresses and dates.
Pick ONE real bottleneck in that industry, written as one sentence, the way you would after discovery.
Build an agent or workflow that removes that bottleneck, using the code, model or human split from lesson 1.
Connect it to real infrastructure: a real Postgres database, and at least one external API with real authentication (an API key or OAuth), not just local files.
Handle edge cases: schemas, validation, retries and a human review path (lesson 2).
Wrap it in an eval harness: a golden dataset of 30 to 100 cases and a pass rate you rerun after every change (module 9).
Deploy it in a container on a cloud service so others can try it, not only on your laptop.
Write it up like an FDE: a scoping document, the trade-offs you made, what broke and how you fixed it, and what you would standardize to run it for ten customers.
Where to find data
Use well-known public sources and always read the licence. Data.gov lists US government datasets on transport, health, finance and more. Hugging Face Datasets and the UCI Machine Learning Repository host many text and tabular datasets. Some sources fit specific briefs well: the Consumer Complaint Database from the US Consumer Financial Protection Bureau has real complaint text with product and issue labels; the CUAD dataset from The Atticus Project has commercial contracts with labelled clauses; ClinicalTrials.gov publishes trial records with eligibility criteria as free text. Real data gives you real mess; synthetic data lets you plant known edge cases for your golden set.
The 30-day plan
Plan about an hour or two a day. Each week ends with something you can show.
Week 1, complete one real loop: the agent loop or workflow, two or three tools, basic guardrails (allowed actions, step limit), the context it needs, simple memory such as case history in Postgres, and an audit trail table. One case should go from input to result end to end.
Week 2, make it recover: JSON schemas for every model output, validation, a failure modes catalogue, exception handling, and a review queue for cases it cannot handle.
Week 3, make it measurable and affordable: retries with backoff, a golden dataset with a pass rate, cheaper models for simple subtasks where evals allow, cost per case, and an estimate of time, risk or revenue effect against a baseline.
Week 4, defend it: deploy it, then rehearse two talks. As the engineer: architecture, decisions and trade-offs. As the sponsor: problem, outcome, evidence, risk. Show it to a few real business people from that industry and write down their feedback.
What the write-up must show
Keep it short and honest: a one-page scoping document (problem, user, success measure, out of scope) and a system diagram. A table of trade-offs, for example 'used a fixed workflow instead of an agent loop because the steps never change'. A 'what broke' section with two or three real failures and fixes. Eval results with the pass rate and the main failure reasons. Finally, 'running it for ten customers': what you would make configurable, what you would monitor, and what onboarding a new customer would take. That last section shows product thinking.
Worked example
Worked example: eight project briefs
Each brief lists the industry, the bottleneck, a data idea, what the agent does, where a person approves, and how you measure it. Pick one, and narrow it further if you can. Check each data source's licence before you use it.
Accounts payable invoice exception triage (finance). Bottleneck: clerks spend hours on invoices that do not match the PO. Data: synthetic invoices, POs and supplier records generated with Faker, with planted mismatches. Agent: extracts fields, runs the match in code, explains the mismatch and drafts a supplier query. Human: approves any hold or supplier email. Eval: correct exception reason on a golden set of planted mismatches.
Insurance claims intake (insurance). Bottleneck: claim forms and photos arrive incomplete and staff chase missing details. Data: synthetic claim forms and emails made with Faker. Agent: extracts claim fields, checks policy rules in code, lists missing items and drafts a request. Human: approves any decline or payment step. Eval: missing-item detection rate and field extraction accuracy.
Clinical trial eligibility extraction with PII masking (healthcare). Bottleneck: coordinators read long eligibility criteria by hand. Data: trial records from ClinicalTrials.gov plus synthetic patient notes. Agent: masks names and identifiers, extracts structured inclusion and exclusion criteria, and flags likely matches. Human: a coordinator confirms every match. Eval: criteria extraction accuracy and zero unmasked identifiers in outputs.
Contract clause review (legal). Bottleneck: lawyers scan every contract for a short list of risky clauses. Data: the CUAD dataset of labelled contract clauses. Agent: finds and quotes target clauses, compares them to a standard playbook and marks deviations. Human: a lawyer accepts or rejects each flag. Eval: clause found and correctly labelled against CUAD labels.
Customer complaint routing with knowledge-base answers (financial services support). Bottleneck: complaints sit in a general queue before reaching the right team. Data: the CFPB Consumer Complaint Database text and labels. Agent: classifies product and issue, routes the case, and drafts a reply from a small knowledge base you write. Human: approves every reply sent to a customer. Eval: routing accuracy against the dataset labels and reviewer approval rate of drafts.
Logistics delay explanation and customer updates (logistics). Bottleneck: customer service cannot quickly explain why a shipment is late. Data: transport datasets found on Data.gov plus synthetic orders and shipment events. Agent: reads shipment events, explains the likely delay cause in plain words and drafts a customer update. Human: approves updates for key accounts. Eval: correct delay cause on planted cases and draft approval rate.
HR policy assistant (HR). Bottleneck: HR answers the same policy questions every day. Data: a set of policy documents you write, modelled on public examples, plus synthetic employee questions. Agent: answers with quotes from the policy and refuses when the answer is not in it. Human: HR reviews any answer about pay, leave disputes or discipline. Eval: answer correctness and correct refusal rate on a golden question set.
Sales call notes to CRM updates (sales). Bottleneck: reps skip CRM updates after calls. Data: synthetic call transcripts you write or generate, with a mock CRM in Postgres. Agent: extracts next steps, deal stage and contacts, and proposes CRM field changes. Human: the rep approves the changes. Eval: field-level accuracy against the expected updates.
Start practicing
Practice 1
Choose your project and write the scope
60 min
Goal: Leave with a chosen project and a one-page scoping document you can start building on tomorrow.
Pick one brief from the example, or write your own using the same six fields.
Open the data source for it and download or view a sample; write down three ways the data is messy.
Write the bottleneck in one sentence and one success measure with how you will compute it.
Classify the workflow steps as code, model or human, using lesson 1.
Write an 'out of scope' list of at least three things you will not build.
Put the four weekly goals from this lesson into your calendar with a concrete deliverable for each.
Done when: You have a one-page scope document, a data sample on your machine, and four dated weekly deliverables.
Practice 2
Stand up the real infrastructure
60 min
Goal: Prove the foundation of your project works before writing any agent code.
Install Docker and run Postgres in a container, following the Docker getting started guide.
Create two tables for your project, for example cases and audit_log, using the PostgreSQL tutorial for syntax.
Install Faker with pip and write a short Python script that inserts 20 synthetic cases.
Get a free API key for one external service your project needs and store it in an environment variable, never in code.
Commit the scripts to a new GitHub repository with a README that states the bottleneck.
Done when: A query on your Postgres container returns 20 synthetic cases, and the repository is on GitHub without any secret in it.
Common mistakes
Building a general chatbot. Pick one bottleneck and one user.
Only using clean sample data. The mess is where FDE skill shows; keep it in.
No eval harness, so you cannot say how well it works. A pass rate on a golden set is the minimum.
Leaving it on your laptop. A deployed link and a short demo video are far easier to share.
A write-up that lists features instead of decisions, failures and trade-offs.
Check yourself
What makes a portfolio project show FDE skills rather than only coding skills?
What should each week of the 30-day plan add?
Why mix public and synthetic data?
What goes in the 'running it for ten customers' section, and why does it matter?
Lesson 8.6
Preparing for FDE interviews
You will know the common shape of FDE hiring loops and have a practice plan for each round, including live Python without an IDE and questions about your own project.
Why this matters on a client engagement
FDE interviews test the job itself: can you write working code, design a system that survives real use, scope a vague client problem out loud, and explain your choices to people who are not engineers. Candidates who prepare only for algorithm puzzles are often surprised. Knowing the usual shape of the loop lets you practice the right things, and the practice makes you better at the real work too.
Common FDE interview rounds
Behavioral
Ownership of an outcome
Handling a hard customer
Explaining tech to non-engineers
Being wrong and changing course
Technical deep dive
OOP Python with state
Coding in a plain editor
Agentic system design
AI security questions
Deep dive on your project
Case study
Vague client problem
Clarifying questions aloud
Bottleneck and success measure
MVP then production plan
Take-home
Clone repo, fix issues
Tests and error handling
Short video: what and why
Judged on production readiness
Each column is one kind of round and what to practice for it.
The common shape of the loop
Loops differ between companies, but candidates commonly report three kinds of round, sometimes with a take-home assignment as well. A behavioral round asks about ownership and communication: a time you owned a problem end to end, handled an unhappy customer, or changed course after being wrong. A technical deep dive covers coding and system design, often agentic system design as in lesson 3, plus questions about AI security. A case study round gives you a vague client problem and asks you to scope it aloud: questions, bottleneck, MVP, success measure, risks.
Expect deep questions about your own projects in any round. Interviewers pick one decision and keep asking why. This is where the portfolio project from lesson 5 pays off, if you built it yourself and wrote down the trade-offs.
Object-oriented Python, for real
OOP questions are rarely 'define polymorphism'. More often you are asked to design 30 to 50 lines of clean classes that hold state and enforce rules: a rate limiter, a ticket queue with priorities, an inventory that cannot go negative, a simple bank account with a transaction history. Interviewers look for clear names, small methods, state that cannot be put into an invalid condition, sensible errors, and a short demo that shows it works.
Know the standard library pieces that make this easy: dataclasses for simple record classes, heapq for priority queues, collections.deque for sliding windows, enum for fixed choices, and typing hints. The example shows a ticket queue in under 40 lines.
Live coding in a plain editor
Some loops ask you to write Python in a shared document or plain text editor, with no syntax highlighting, no autocomplete and no way to run it. The point is to see whether you can write correct code from memory. Know your imports exactly (from dataclasses import dataclass, field; import heapq), write real code rather than pseudocode, and trace through it by hand with a small input before saying you are done. Talk while you write: say what each class is for and what case you are handling.
Take-home assignments are also common. A typical form is: clone a repository, fix a few issues, and record a short video explaining what you changed and why. They are judged on production readiness (tests, error handling, clear commits, no secrets in code) and on communication. A clear five-minute video that explains trade-offs often matters as much as the code.
AI security and soft skills
Be ready to explain prompt injection (instructions hidden in data the model reads, such as an email that says 'ignore your rules'), jailbreaks (users trying to make the model break its rules), PII protection (masking, access control, retention), and how the system behaves when the model fails or is unavailable. Good answers name concrete controls: treat retrieved content as data, narrow tool permissions, validate outputs, human approval for risky actions, and a manual fallback.
Soft skills are tested throughout, not only in the behavioral round. Interviewers watch for customer empathy, ownership of the outcome rather than the task, breaking a vague problem into parts, product sense (what is worth building), comfort with ambiguity, and explaining a technical decision to a non-engineer in plain words. Prepare five short stories from your own experience that show these, each with the situation, what you did, and the result.
Worked example
Worked example: a ticket queue with priorities
This is the kind of class design an interviewer might ask for: 'Build a support ticket queue. Urgent tickets come first; tickets with the same priority are served in arrival order; the same ticket must not be queued twice.' It runs as written with Python 3.11 and prints ['T2', 'T3', 'T1'] None.
Points to say aloud while writing it: the Ticket dataclass orders by rank then by an increasing sequence number, which keeps equal priorities first in, first out; the queue rejects unknown priorities with a clear error; a set of open ids makes add idempotent; pop returns None when empty instead of raising. Then mention what you would add next: a peek method, tests with unittest, and a lock if several threads use it.
python
import heapq
import itertools
from dataclasses import dataclass, field
PRIORITY = {"urgent": 0, "high": 1, "normal": 2, "low": 3}
@dataclass(order=True)
class Ticket:
rank: int
seq: int
ticket_id: str = field(compare=False)
title: str = field(compare=False)
class TicketQueue: # most urgent first; equal priority is first in, first out
def __init__(self) -> None:
self._heap: list[Ticket] = []
self._counter = itertools.count()
self._open: set[str] = set()
def add(self, ticket_id: str, title: str, priority: str = "normal") -> None:
if priority not in PRIORITY:
raise ValueError(f"unknown priority: {priority}")
if ticket_id in self._open:
return # same ticket sent twice: ignore it
self._open.add(ticket_id)
heapq.heappush(self._heap, Ticket(PRIORITY[priority], next(self._counter), ticket_id, title))
def pop(self) -> Ticket | None:
if not self._heap:
return None
ticket = heapq.heappop(self._heap)
self._open.discard(ticket.ticket_id)
return ticket
q = TicketQueue()
for tid, title, prio in [("T1", "Printer jammed", "low"), ("T2", "Payroll failed", "urgent"),
("T3", "VPN slow", "normal"), ("T2", "Payroll failed", "urgent")]:
q.add(tid, title, prio)
print([q.pop().ticket_id for _ in range(3)], q.pop())
Start practicing
Practice 1
Plain-editor coding drill
45 min
Goal: Write correct, runnable Python without highlighting, autocomplete or running it first.
Open a plain text editor or an empty online document with spell check off. Set a 25-minute timer.
Write a RateLimiter class that allows at most N calls per T seconds per user id, with an allow(user_id, now) method and a short demo at the bottom.
Write every import exactly. Do not look anything up and do not run the code yet.
When the timer ends, trace the demo by hand on paper and write the output you expect.
Copy the code into a .py file and run it with python3. Note every error and every difference from your expected output.
Repeat tomorrow with a different class, such as an inventory that cannot go negative, and compare your error count.
Done when: Your second attempt runs first time with the output you predicted by hand.
Practice 2
Project deep dive rehearsal
30 min
Goal: Be able to defend your own project under repeated 'why' questions.
Write a two-minute summary of your portfolio project for an engineer and a separate one for a business sponsor.
List five decisions you made and the alternative you rejected for each.
Paste the summary into a free LLM chat and ask it to act as an FDE interviewer and ask 'why' five times on one decision.
Answer aloud, recording on your phone, then note any answer where you were vague.
Done when: You have two summaries, five decisions with rejected alternatives, and a recording where you answer five follow-up questions without guessing.
Common mistakes
Preparing only algorithm puzzles. Practice class design, system design and scoping aloud as well.
Writing pseudocode in a live round when runnable code was asked for. Know your imports.
Talking about a project you cannot defend in depth. Only present work you built and understand.
Jumping to a solution in the case study without asking questions first.
Giving vague security answers. Name the threat and the concrete control.
Check yourself
What three kinds of round do candidates commonly report, and what does each test?
What do interviewers look for in an OOP design question beyond correct syntax?
How do you prepare for writing code in an editor with no autocomplete?
How would you explain prompt injection and one defence to a non-engineer?