Stop Trusting Your AI Agent. Start Testing It.

Your AI agent sounds confident. That is not the same thing as being right.

The demo looked perfect. Then a real customer typed something weird.

Elena runs a five-person dental clinic in Manila. Her front desk was drowning in the same messages every day. What time do you open on Saturday. Do you take this insurance. Can I move my cleaning to next week. So she did what a lot of owners are doing in 2026. She wired up a chatbot on WhatsApp and the clinic website, pointed it at Claude, and watched it answer five questions perfectly in a row.

Then a patient asked if the clinic offered a discount for paying in cash, phrased as a joke buried inside a real scheduling question. The bot answered the joke as if it were policy. Nobody caught it until a second patient asked for the same "discount" a week later.

Nothing about that bot was broken. It was never tested. Those are two different problems, and most small businesses only ever solve the first one.

Right now, "testing" an AI agent means typing a few questions and hoping

Here is how most owners actually check an agent before it goes live:

• Type three or four questions you can think of off the top of your head.
• Read the answers. They sound fine.
• Ship it.

Three questions is not a test. It is a vibe check. Real customers will ask the same question four different ways, in the wrong order, with typos, in a bad mood, at 11 p.m. A handful of tidy questions from you will never surface that.

You are not testing your agent. You are hoping it behaves. And hope is a bad plan for the moment a customer asks about a refund, a price, or a medical detail.

What just shipped: an AI agent that writes its own test suite

On 22 July 2026, LangChain, the company behind the widely used LangChain and LangGraph agent frameworks, released something called the Eval Engineering Skill. It is a free, open-source skill you install directly into Claude Code or OpenAI's Codex. Point it at the folder holding your agent, and it reads the agent's prompts, tools, and any real conversation logs you give it. Then it proposes a list of specific abilities worth testing, for example "does it correctly decline to quote a discount that does not exist," and asks you to approve or adjust each one.

Once you approve, it writes the tests as runnable tasks in a format called Harbor, hands them to the agent, records exactly what happened, and grades whether the agent actually did the right thing, not just whether the answer sounded confident. Run the same tests again after you change a prompt, swap in a cheaper model, or add a new tool, and you immediately see whether you made things better or worse.

That is genuinely new. Testing an AI agent used to mean hiring someone who understood evaluation frameworks. Now the agent building the test suite is the same kind of tool, Claude Code or Codex, that a lot of small teams already use to build the agent in the first place.

"Isn't this just for developers?" No, and here is the plain version

You do not need LangChain's skill, or even a developer, to run this same idea today. A Manila clinic owner, a Lagos retailer, or a solo bookkeeper in Manchester can do the exact version of this with tools that already exist. Alfred Nutile, who runs the channel DailyAi.Studio, walked through this live for two very ordinary small-business cases: a chatbot for a sports club, and an email automation that reads unstructured messages and pulls out structured data.

His method skips the jargon entirely. You write down, in plain English, what the agent should and should not do. Then you open Claude Code, point it at that plain-English file and a batch of real sample messages, and say one sentence: use Promptfoo to build the prompt and test it against this data until it works close to 100 percent of the time. Claude Code does the technical part. You supply the business judgment.

How to build a real confidence test this week

You do not need Harbor, LangSmith, or any enterprise tooling to start. Here is the version any owner can run with Claude Code, Codex, or even a good general-purpose AI assistant this week.

1. Write the plain-English spec. One page. What is this agent for. What must it never say (discounts that do not exist, medical or legal advice, a promise you cannot keep). What counts as a good answer. No code, no prompt engineering, just the business rules you already know by heart.
2. Pull 50 to 200 real messages. Actual WhatsApp chats, support emails, booking requests, whatever your customers actually send. Include the messy ones: typos, slang, an angry customer, two questions crammed into one message.
3. Hand both files to your AI coding assistant. Tell it plainly: "Use this spec and this sample data to build and test the prompt for this agent until it answers correctly almost every time." Let it pick the tool, whether that is Promptfoo, the Eval Engineering Skill, or something simpler.
4. Read the score, not just the vibe. A real test run reports a number: 14 out of 14 correct, then 42 out of 42 across three runs. If it is not climbing toward that, the agent is not ready, no matter how smooth any single answer sounded.
5. Test with a second model as the judge. Grading your agent's answers with the exact same model that generated them is like asking a student to mark their own exam. Swap in a different model, even a free local one through a tool like LM Studio running something like Gemma, purely to check the work.
6. Rerun the whole suite before every change. New prompt, new model, price increase, new menu, new policy. Rerun the tests. If the score holds, ship it. If it drops, you just caught the mistake before a customer did.

The second example: turning messy emails into data you can trust

The same method works for automations that never touch a chat window at all. Say a clinic, an agency, or a landlord needs to pull structured details out of unstructured emails: a name, a service requested, an insurance provider, a reply address that is not always the same as the "from" address. That is exactly the second case Nutile ran live. He gave the agent a folder of sample emails with the correct answers already labelled, then asked Claude Code to build an extraction prompt and prove it against every one of those emails before it touched a single real message.

What changes in a week, a month, six months

In a week, you have one agent, probably your support or booking chatbot, with an actual spec file and a test suite behind it, instead of a vibe check. You will already have caught at least one wrong answer you did not know it was giving.

In a month, every new automation you build, whether it answers a WhatsApp message, drafts a quote, or reads an inbox, starts with the same two files: the plain-English spec and a batch of real sample messages. Testing stops being a separate, scary step and becomes the first thing you do, not the thing you skip.

In six months, you are not the person anxiously reading every AI-generated reply before it goes out. You are the person who reads a confidence report once a month and decides whether to expand what the agent is trusted to do next. Same team, same tools, a business that can actually say why it trusts its own AI agent.

You already know what a good answer to your customer looks like. Let the AI figure out how to get there, and then make it prove it, before a real customer finds the gap for you. If you have already read our piece on agents that can test and fix their own work, this is the step that comes first: decide what "working" even means, in writing, before any agent gets to grade itself.