Exercise, baseline versus configured

The lessons of this module described what a harness is. This exercise lets you measure what one part of it changes, on your own machine, with the agent you already use.

It is also the first run of a comparison that comes back in module 8 under the name Swift Agent Harness Benchmark. There, three configurations are compared. A baseline, a configured environment and an extended, governed one. Today you run the first two.

Configuration	What it has
A. Baseline	The agent, a clearly specified task, and nothing else in the repository
B. Configured	The same, plus repository instructions, a verification command and stop conditions
C. Extended and governed	B plus custom tools, execution controls, policy enforcement and automated validation. Module 8.

What you need

• The course repository, which contains the AgenticShop codebase and the starter files of this exercise. Its address is given on the course page once the repository is published.
• A coding agent. Claude Code and Codex both work.
• About an hour.

Check that the baseline builds before you start.

cd agenticshop
swift build
swift test

The feature

Customers need to find a product by typing part of its name. The feature belongs to the Catalog module and works with any ProductCatalog.

Id	Requirement
SEARCH-1	An empty query, or a query made of spaces only, returns every product.
SEARCH-2	Matching ignores case and diacritics. “cafe” finds “Café Press”.
SEARCH-3	A product matches when the query is contained in its name, or equals one of its tags.
SEARCH-4	Results keep the order of the catalog.

Run A, baseline

1. Create a branch named run-a from the tag baseline.
2. Start a new agent session in agenticshop. Do not add any instruction file.
3. Type one prompt and nothing else.

Add product search to the catalog.

4. Accept what the agent proposes. Answer its questions briefly, and count them.
5. When it says it is done, run swift build and swift test yourself.

Run B, configured

1. Create a branch named run-b from baseline. Run A must not be visible from it.
2. Copy the file starter/harness/AGENTS.md of the exercise to the root of agenticshop. It states the build and test commands, the conventions of the codebase and what counts as done.
3. Start a new session. Give the agent the content of starter/harness/task.md, with the four requirements pasted in. The task states a scope, a verification command and the conditions under which the agent must stop and report.
4. Let it work. Count your interventions.
5. When it stops, run swift build and swift test yourself.

Use a new session for each run. A session that has seen run A already knows the answer.

Record both runs

Fill in the table from what you observed. The template is starter/comparison-report.md.

Criterion	Run A	Run B
swift build after the run
swift test after the run
Tests written by the agent
Requirements covered by a test, out of 4
Files changed outside Packages/Catalog
Questions the agent asked
Times you had to correct it
Elapsed time

Then list each difference, and name the dimension you attribute it to. Most will land in context engineering, feedback and validation, or governance. Say which, and say why.

Read your result

A few things to look for when you compare the two branches.

SEARCH-1 says that a query of spaces counts as empty. Nothing in the one-line prompt of run A mentions it. See whether run A handled it, and if run B did, where it learned it.

Look at what “done” meant in each run. In run A the agent decided on its own. In run B a command decided. That is the difference between delivering a claim and delivering evidence.

Look at what did not change. If both runs produced clean, similar code, that is a result too. A harness does not make the model write better Swift. It changes what the model knows and what checks its work.

Imagine opening each branch tomorrow in a new session. See which one explains itself, through its tests, its commits and what it left behind.

One run is one sample. Agents vary from one run to the next. If you have time, repeat each run once and see whether your conclusion holds. The benchmark in module 8 repeats every mission for that reason.

Going further

Two optional experiments, if you want more than one comparison.

Remove one part at a time. Start from configuration B and take away a single element. First the instruction file, then the verification command, then the stop conditions. Run the task after each removal. The removal that hurts most shows which part carries the most weight for this task. It does not show where your own project is weakest. For that you need the failure log from the lesson on the eight dimensions.

Measure your verification gap. Over your next five tasks, note each time the agent says it is done, then check with tests it did not write. Count how often the claim was wrong.

Done when

• Your report compares both runs on the same criteria.
• Each difference is attributed to a named dimension.
• The feature builds and its tests pass in run B, or your report explains why not.

A reference implementation of the feature is available with the course solutions. It contains no measurements, because the ones that count are yours.