Retrieval supplies an AI system with material at the moment it handles a request, so it can use current facts. Training changes the model itself, shaping how it behaves and decides. Most real assistants use both. Before preparing any data, define the task, the evidence it needs and the actions it may take, then give each source a clear job.

The assistant that quoted the rule and broke it

Picture a support assistant at a mid-sized retailer. A customer asks for a $1,400 refund. The assistant searches the knowledge base, finds the policy that says refunds above $1,000 need a manager's approval, quotes it word for word, and then issues the refund anyway.

Nothing was missing from its documents. The assistant had the rule. What it lacked was the behavior: noticing that this request crossed a line, stopping, and routing it to someone with authority. If you graded only the answer text, it passed. If you checked the payment system, it failed.

That gap is the clearest way to understand a question many teams get wrong: should we train the model on our data, or just give it a knowledge base?

Two different jobs that get confused

Retrieval hands the system material at the moment it handles a request. The model reads the current refund policy the way a new hire reads a binder. Update the policy on Monday and the assistant can use the new version on Monday.

Training changes the model itself. Worked examples shape how it behaves: when it asks a clarifying question, when it escalates, how it weighs a messy request against a clean rule. That learning persists, which is why it suits judgment and is a poor home for facts that change every quarter.

Most real deployments combine a trained model, retrieval, tools and hard application rules. The useful question is not which technique wins. It is which part of the job each one should carry.

·         Facts that change often: retrieve them from an approved source.

·         Judgment that repeats: teach and test it with connected examples.

·         Actions with consequences: enforce limits in the application, then test that the model respects them.

Give your data a job description

Before anyone exports a folder, write one sentence about what the assistant must do. “Answer questions about our return policy” is a retrieval job. “Handle refund requests the way our best agent would, including the ones that need a manager” is a behavior job, and it needs very different evidence.

For the behavior job, a clean policy document is not enough. You need connected cases: the request, what the agent could see at the time, the decision, the approval when one was required, and what happened next. Those threads show what a correct next step looks like. Sometimes the correct step is to escalate rather than finish.

One trap to avoid: keep notes written after the case closed out of any test input. A resolution summary is great for learning, but in a test it simply hands over the answer. For every task, spell out four things:

·         The request or action the system must handle.

·         The approved information available at that moment.

·         The actions it may take, and who must approve the rest.

·         The evidence you will use to judge the result.

Old decisions are evidence, not an answer key

It is tempting to treat years of support history as a ready-made set of correct answers. It isn't. A past agent may have skipped the approval too. A case marked “closed” may only mean someone stopped replying.

Ask someone who knows the work to review the labels, and write down where reviewers disagree. Keep source values separate from labels added during preparation. When context is missing, say so instead of filling the gap with a confident reconstruction.

If cases are related, such as one customer writing in three times, keep them on the same side of the line between the examples you develop with and the examples you test on. Otherwise the test leaks its own answers.