An AI agent is a software system that uses a language model to plan and complete multi-step tasks by calling tools, reading data and deciding what to do next. This article explains what an agent is made of, the four points where teams lose the ability to explain its behaviour, and the six steps we follow to build agents that stay inspectable in production. It is written for product and engineering teams preparing to move an agent out of the prototype stage.
Nearly every agent project we are asked to review works beautifully in a demo and becomes unexplainable within weeks of going live. We see the same sequence every time. Someone builds a convincing prototype, the business asks for it in production, and then the first support ticket arrives: why did it do that? Nobody can answer, because the decisions that mattered were never recorded.
We build agents across retail, logistics and enterprise operations. What separates the ones that survive contact with real users from the ones that get quietly switched off has very little to do with model choice. It comes down to whether the team can reconstruct what happened on any given run.
What is an AI agent?
An AI agent is a system that pairs a language model with tools, memory and an orchestration layer so it can pursue a goal across several steps rather than answering a single prompt. The AI agent meaning matters here, because the term now covers everything from a scripted chatbot to a system with write access to your ERP.
Four components do the work:
- The model. Handles reasoning and language. It is the part teams obsess over and the part that causes the fewest production problems.
- Tools. The functions the agent can call: APIs, database queries, search, calculations, actions in other systems.
- Context and memory. What the agent knows at each step, including retrieved documents, conversation history and system state.
- Orchestration. The rules governing the loop. When the agent runs, when it stops, what it is allowed to touch, and when it hands over to a person.
What is an AI assistant, then? We use that term for the interface a person talks to. An assistant may have an agent underneath, or it may just be a well-designed prompt. The distinction is worth keeping, because the oversight requirements differ completely.
Where agents become black boxes
Traceability disappears at four specific points, and in our experience all four are lost before the system reaches production rather than after.
Tool selection. The model decides which function to call. If you only log the outcome, you never learn which alternatives it considered or why it picked one.
Context assembly. The prompt sent to the model on step three of a run is rarely the prompt anyone wrote. It is assembled from retrieval results, prior steps and system state. Teams log the template and lose the actual input.
State between steps. Agents carry information forward. When something goes wrong on step five, the cause is usually a value that entered on step two.
Escalation. Handovers to a human are often the least instrumented part of the system, which means the cases most worth learning from are the ones you have the least data on. The OWASP guidance on language model risks is a useful starting point for what else to watch.
How to create an AI agent: our six steps
We build agents in six stages, and each stage produces an artefact that can be audited later. The sequence matters more than the tooling.
1. Define the task boundary. Write down what the agent is allowed to do and what it must refuse. A narrow agent that handles one workflow reliably is worth more than a general one that handles six unpredictably.
2. Map the tools and the data. List every function the agent can call and every data source it can read, with the permissions attached to each. This list is your risk surface, and it should fit on one page.
3. Build the context layer before the agent. This is the step teams skip, and it is the one that decides whether the agent works. When we built the product locator for Maxeda DIY Group, the conversational layer answering “where can I find wall plugs?” came last. First we created a dynamic database of over 17 million unique product-location combinations across more than 330 stores, integrated with SAP, and gave store teams a scanner app to define aisle labels in their own words. An agent is only as explainable as the data it sits on.
4. Make the loop observable. Log every step: input, tool call, tool response, model output, decision. Use structured tracing rather than text logs. OpenTelemetry conventions work well here. This is where we use Sinas, the open-source orchestration and control layer built by our AI technical lead Kjeld Oostra, which connects data, agents and tools into secure, traceable workflows. Its API Connectors handle the tool layer, and because the whole thing is open source the trace stays portable instead of locked inside a vendor platform.
5. Evaluate against real cases. Assemble fifty to a hundred real requests, including the awkward ones, and run them on every change. Anthropic’s engineering notes on effective agents make the same argument for starting simple and measuring before adding complexity.
6. Design the escalation path. Decide in advance which cases go to a person, how the agent hands over, and what the person sees. Under the EU AI Act’s transparency obligations, for many use cases this is a requirement rather than a nicety.
AI agents examples: three patterns we build most often
Production agents tend to fall into three patterns, and each carries a different oversight requirement. Choosing the pattern deliberately at the start saves a rebuild later.
| Pattern | What it does | Autonomy | Oversight needed | Typical failure |
|---|---|---|---|---|
| Retrieval agent | Answers questions from your own data | Low, read only | Source citation on every answer | Confident answers from stale or partial data |
| Workflow agent | Completes a defined process end to end | Medium, writes within limits | Step logging plus approval on write actions | Silent partial completion halfway through |
| Supervisory agent | Coordinates other agents or long-running tasks | High | Full trace, budget caps, mandatory escalation | Loops, runaway cost, drift from the original goal |
Our advice is to start with the first pattern and earn your way to the third. We rarely see a supervisory agent succeed in an organisation that has not already run a workflow agent in production for a few months. The NIST AI Risk Management Framework is a solid reference for setting the controls at each level.
Key takeaways
- An AI agent combines a language model with tools, context and orchestration to complete multi-step tasks, and the model is the smallest part of that system.
- Traceability is lost at four points: tool selection, context assembly, state between steps, and escalation to a human.
- The data and context layer should be built before the agent itself, because an agent can only be as explainable as the data underneath it.
- Structured tracing of every step, using open standards rather than a closed vendor platform, is what makes an agent auditable after the fact.
- Start with a read-only retrieval agent, prove it in production, and add autonomy only once the oversight is working.
Ready to build an agent you can actually explain?
We help teams design, build and deploy agents that hold up in production and in an audit. If you are planning your first one, or trying to work out why the one you have behaves unpredictably, talk to our team about where to start.
