Knowledge for agents: retrieval that can iterate

Most retrieval writing is about RAG: retrieval-augmented generation, the pattern where a chat app fetches a few relevant chunks and stuffs them into the prompt before answering. Agents change the picture enough that copying chat-RAG wholesale is a mistake. The difference is one capability, and this chapter is about what follows from it: a chat model gets one retrieval and must answer from it; an agent gets a retrieval tool it can call repeatedly, reformulating and verifying as it goes. That single shift moves work out of the retrieval pipeline and into the loop, and it changes chunking, ranking, and even whether you need vectors at all.

Chat RAG versus agentic retrieval

Set them side by side:

chat RAG:   query --> retrieve top-k --> stuff into prompt --> answer
            (one shot; if retrieval missed, the answer is wrong)

agentic:    the loop, with a search tool
            model: search("database timeout")      -> weak results
            model: search("connection pool v841")  -> better
            model: grep("INC-2091")                -> the exact ticket
            model: reads, decides it needs more, searches again
            model: answers, citing what it actually read

The chat pipeline must be good in one shot, because there is no second try; enormous effort goes into chunking strategy, re-ranking, and query expansion to make that single retrieval land. The agent needs its retrieval to be good enough to iterate from, because a weak first result is not a failure, it is information: the agent reads it, learns the right vocabulary, and searches again. This is the same tool-in-a-loop machinery from Part 1, pointed at a knowledge base, and it is why the previous chapter insisted an agent wants both grep and vectors: the loop chooses per call which retrieval mode fits the sub-question it has right now.

What changes when the model can iterate

Three design consequences fall out, each a place where agentic retrieval should look different from chat RAG:

• Chunking can be coarser. Chat RAG chunks small to fit many precise fragments in one shot; an agent can retrieve a coarse chunk, read it, and drill down with a follow-up search, so over-fragmenting (which severs context across chunk boundaries) hurts more than it helps. Retrieve documents or sections, let the agent navigate within them.
• Perfect ranking matters less; recall matters more. Getting the single best chunk to position one is a chat obsession because position one is all the model sees. An agent scans several results and picks; what wrecks it is the right document not being in the candidate set at all. Tune for recall (is it in the top twenty?) over precision-at-one.
• Retrieval becomes a tool decision, not a pipeline stage. The agent decides when to search, what to search, and which mode (keyword, vector, hybrid) to use, which means the leverage moves to the tool descriptions: "use grep for exact ids and error codes; use search for concepts and themes" steers retrieval quality more than a re-ranker would. The best retrieval improvement is often a better tool description, not a better index.

Where Knowledge Bases fit

Bedrock Knowledge Bases is the managed retrieval pipeline: point it at documents in S3, it handles ingestion, chunking, embedding, and a vector store (increasingly S3 Vectors underneath), and exposes a retrieve API plus richer variants (GraphRAG, which builds a graph over entities for multi-hop questions; structured retrieval, which translates natural language to SQL over your warehouse). Decoded through this chapter's lens, a Knowledge Base is a very good chat-RAG pipeline you can rent, and that is exactly how to use it in an agent: as one tool among several, not as the whole retrieval story. The agent calls it for "search the docs," alongside a grep tool for exact lookups and whatever domain tools the task needs, and the loop composes them. Renting the pipeline is a fine decision; letting the rented pipeline dictate a one-shot, chat-shaped retrieval flow onto an agent that could iterate is the trap.

Grounding and its limits

Retrieval is also how agents stay grounded in fact rather than confabulating, and the honest scope matters. Retrieved evidence, cited by event id so a claim traces to the memory or document that supports it, is the substrate that makes the verification mesh possible: a verifier can check a claim against its cited source. But retrieval grounds only what it retrieves. An agent that never searched, or searched badly, answers from the model's parametric memory with full confidence and no evidence trail, and no amount of vector-store quality fixes a retrieval the agent chose not to do. This is why the retrieval tool description and the model's disposition to use it (a Part 2 steering concern) matter as much as the index: grounding is a behavior before it is a pipeline.

Don't be confused: knowledge vs memory. They share the vector machinery and blur in casual use, but the sources differ and it changes their handling. Knowledge is external and authored: documents, wikis, code, tickets, ingested and retrieved, and the agent does not write it. Memory (Chapter 35) is internal and earned: facts the agent distilled from its own experience, which it does write, and which therefore needs provenance and staleness policing that an authored document already has. Retrieving from a stable knowledge base and retrieving from a self-updating memory are the same query over different trust profiles.

👉 Next: long-term memory pipelines, where the agent's own experience becomes durable memory, and the harness that answers "is this memory system actually any good" gets built and run.