Tool design: the contract that steers the agent

The loop treats a tool as a name and a function. The model sees something richer: a name, a description, and a parameter schema, and it chooses what to do next by reading them. That makes tool definitions a strange and powerful kind of code: prose that executes, in the sense that changing a sentence changes the system's behavior. This chapter is about writing that prose and its schema deliberately, and it ends with a lab that puts a number on the difference.

Anatomy of a definition

Here is query_metrics as the model receives it, in full:

{
  "name": "query_metrics",
  "description": "Read p99 latency by hour for one service. Use when the question is how slow or how many.",
  "input_schema": {
    "type": "object",
    "properties": {
      "service": {"type": "string"},
      "window":  {"type": "string"}
    },
    "required": ["service", "window"]
  }
}

Three parts, three different jobs:

• The name is an identifier the model will emit verbatim. Specific beats generic (query_metrics over get_data) because the name is also a hint: it is read as part of the decision, not just quoted back.
• The description answers one question above all: when should I be called? We will spend most of the chapter here.
• The input schema defines the argument shape, written in JSON Schema, a standard language for describing what a piece of JSON must look like: this field is a string, these fields are required, this one only allows three values. The model generates arguments against the schema, and the loop can validate against it before executing.

Don't be confused: description vs schema. The schema constrains what the arguments look like once the model has decided to call the tool. The description steers whether and when it decides to call it at all. Teams polish schemas (they look like code) and neglect descriptions (they look like comments), which is exactly backwards for reliability: most wrong-tool behavior traces to descriptions, not schemas.

What a good description contains

Study the difference between these two, both honest:

"Search logs."

"Full-text search over application log lines. Use when hunting for
 error messages, exceptions, stack traces, or specific log text."

The second does three things the first does not. It names the domain precisely (application log lines, not "information"). It carries a when clause: an explicit trigger condition, which current models follow remarkably literally; the guidance shared by model vendors and the managed platforms agrees on this point, so treat "Use when..." as a first-class field that happens to live inside a sentence. And it scatters the vocabulary a user's request will actually contain: error, exception, stack trace. Selection is, at bottom, a matching problem between the request and the descriptions; give the match something to grip.

The same thinking extends into the schema. Closed sets should be enums ("unit": {"enum": ["celsius", "fahrenheit"]}) so invalid values are unrepresentable rather than merely discouraged. Every property gets its own description. required stays minimal, with defaults doing the rest. The rule underneath all of it: push errors from runtime into the definition. A constraint the schema can express is a bug the model cannot write.

Lab 1.2: schema surgery, measured

Claims about description quality deserve a measurement, so here is the lab's trick. A real model chooses tools by reading requests against descriptions; our lab replaces the model with a deliberately dumb stand-in, a selector that scores each tool by plain word overlap between the request and the tool's name plus description, highest score wins, alphabetical order breaking ties. No intelligence whatsoever.

Why is that a fair experiment? Because of the direction of the argument. We are not claiming the stand-in behaves like Claude. We are demonstrating that the descriptions carry the signal: if even a bag-of-words selector routes twelve requests perfectly once descriptions are sharp, then the descriptions were doing the work, and a vastly smarter reader can only do better. And when descriptions are vague, watch what any selector, dumb or brilliant, is reduced to.

Two toolsets cover the same three jobs (read metrics, search logs, list deployments). The vague set says things like "Get data about the system." The sharp set is the style argued above. Twelve fixed requests, each with a known correct tool:

python3 tool_picker.py

toolset 'vague': 5/12 routed correctly (41%)
  miss: 'show p99 latency for checkout overnight' -> check_changes (wanted get_data)
  miss: 'how many 500 errors per minute are we serving' -> check_changes (wanted get_data)
  miss: 'was there a traffic spike after midnight' -> check_changes (wanted get_data)
  miss: 'which service got slower this week' -> check_changes (wanted get_data)
  miss: 'search for stack traces mentioning payments-db' -> check_changes (wanted find_info)
  miss: 'grep the log lines for pool exhausted' -> check_changes (wanted find_info)
  miss: 'any exceptions right after the release' -> check_changes (wanted find_info)

toolset 'sharp': 12/12 routed correctly (100%)

Same requests, same selector. Only the descriptions changed.

The misses are more instructive than the score. Look at how the vague set fails: seven requests all route to check_changes, including "show p99 latency," which has nothing to do with changes. The mechanism: the vague descriptions share no vocabulary with any request, so every request scores zero against every tool, and the decision falls through to the tie-break, where whichever name sorts first eats everything. With nothing to grip, selection is not wrong so much as arbitrary, and arbitrary at scale looks like an agent with a bizarre fixation on one tool. If you have ever watched a real agent inexplicably favor one tool of five, you have probably seen the polished version of this failure.

The sharp set gives the matcher purchase on every request, and even bag-of-words goes twelve for twelve. Full source at the end of the chapter.

Errors are messages to the model

The loop already turns tool failures into strings rather than exceptions. Tool authors decide whether those strings help. Compare:

"error: KeyError"

"error: unknown service 'chekout'. Valid services: checkout, payments,
 search. Check the spelling and call query_metrics again."

The second is written for its actual reader, which is the model on the very next turn. It states what failed, what would have succeeded, and what to do now, and a capable model recovering from it looks like self-correction. As a rule: a recoverable error should read like a helpful colleague; only genuinely fatal conditions (permission denied, resource gone) should read like a wall, and even walls should say which wall. Part 6 leans on this hard, because in a fleet, tools that fail helpfully are the difference between self-healing work units and poison tasks.

Effectful tools carry an operation id

Our three tools only read. The moment a tool writes (files a ticket, sends a message, restarts a service), the delivery realities from the concepts chapter apply: loops crash and retry, queues deliver at least once, so the same tool call may execute twice. The defense is baked into the schema, not bolted on later: effectful tools take a caller-supplied operation_id, and the implementation makes the call idempotent, executing once per id and returning the recorded outcome on repeats.

{
  "name": "file_ticket",
  "description": "File a ticket ... Use only after evidence is assembled.",
  "input_schema": {
    "type": "object",
    "properties": {
      "operation_id": {"type": "string",
        "description": "Unique id for this action; retries reuse it."},
      "title":    {"type": "string"},
      "severity": {"enum": ["low", "medium", "high"]}
    },
    "required": ["operation_id", "title", "severity"]
  }
}

This is gate 5 of the production bar showing up at the smallest possible scale: exactly-once effects on top of at-least-once attempts, arranged per tool.

When a tool deserves to exist

Last question of the chapter: which actions get promoted to dedicated tools at all? There is a tempting shortcut with the opposite philosophy: give the agent one run_shell_command tool and let it do anything. Broad leverage, and every action arrives at your loop as an opaque string. A dedicated tool, by contrast, gives the harness a typed, named hook it can reason about. That trade is the decision rule:

The honest default for real systems: start narrow with dedicated tools, add a sandboxed escape hatch only when breadth is genuinely required, and notice that each row of that table is a gate from Chapter 3 wearing tool-sized clothes.

Full source

"""Lab 1.2: schema surgery. Same requests, same selector, two toolsets.

A real model chooses tools by reading their names and descriptions. This
lab makes that mechanism visible with a deliberately dumb stand-in: a
selector that scores each tool by word overlap between the request and the
tool's name-plus-description, and picks the highest score (alphabetical
order breaks ties, so everything is deterministic).

The stand-in is much cruder than a model, and that is the point: if even a
bag-of-words selector routes correctly once the descriptions are sharp,
the descriptions were doing the work. Vague, overlapping descriptions give
any selector (dumb or brilliant) nothing to grip.

Standard library only. Deterministic: same run, same numbers.
"""

from __future__ import annotations

import re

STOPWORDS = {"a", "an", "the", "for", "of", "to", "in", "on", "and", "or",
             "is", "are", "was", "were", "any", "we", "our", "with", "use",
             "when", "this", "that", "you", "need", "about", "what", "did",
             "who", "how"}


def words(text: str) -> set[str]:
    return {w for w in re.findall(r"[a-z0-9]+", text.lower())
            if w not in STOPWORDS}


def pick(request: str, toolset: list[tuple[str, str]]) -> str:
    """Return the name of the tool whose name+description best overlaps."""
    req = words(request)
    scored = sorted(
        ((len(req & words(name + " " + desc)), name)
         for name, desc in toolset),
        key=lambda pair: (-pair[0], pair[1]),
    )
    return scored[0][1]


# Two toolsets for the same three jobs: read metrics, search logs,
# list deployments. Index in the list encodes the job (0, 1, 2).

VAGUE = [
    ("get_data", "Get data about the system."),
    ("find_info", "Find information you need."),
    ("check_changes", "Check things that happened."),
]

SHARP = [
    ("query_metrics",
     "Read numeric time-series performance metrics such as p99 latency, "
     "error rate, and traffic per service. Use when the question is how "
     "slow, how many, or how much."),
    ("search_logs",
     "Full-text search over application log lines. Use when hunting for "
     "error messages, exceptions, stack traces, or specific log text."),
    ("list_deploys",
     "List recent code deployments and config releases with timestamps "
     "and change notes. Use when asking what shipped or changed recently."),
]

# (request, index of the correct job: 0=metrics, 1=logs, 2=deploys)
REQUESTS = [
    ("show p99 latency for checkout overnight", 0),
    ("how many 500 errors per minute are we serving", 0),
    ("was there a traffic spike after midnight", 0),
    ("which service got slower this week", 0),
    ("find TimeoutError messages for checkout", 1),
    ("search for stack traces mentioning payments-db", 1),
    ("grep the log lines for pool exhausted", 1),
    ("any exceptions right after the release", 1),
    ("what shipped to production yesterday", 2),
    ("list config releases around 2am", 2),
    ("did a deployment change the connection pool settings", 2),
    ("show the change notes for checkout v841", 2),
]


def evaluate(label: str, toolset: list[tuple[str, str]]) -> None:
    names = [name for name, _ in toolset]
    misses = []
    for request, want in REQUESTS:
        got = pick(request, toolset)
        if got != names[want]:
            misses.append((request, got, names[want]))
    right = len(REQUESTS) - len(misses)
    pct = 100 * right // len(REQUESTS)
    print(f"toolset '{label}': {right}/{len(REQUESTS)} routed correctly "
          f"({pct}%)")
    for request, got, want in misses:
        print(f"  miss: {request!r} -> {got} (wanted {want})")


if __name__ == "__main__":
    evaluate("vague", VAGUE)
    print()
    evaluate("sharp", SHARP)
    print("\nSame requests, same selector. Only the descriptions changed.")

👉 Next: the event store. The loop now behaves well and its tools are honest, but everything it does still evaporates when the process exits. Time to give the agent a ledger.