When you test a normal feature, the same input gives the same output. Click the button, check the result, done. AI features don't behave that way. Ask the same question twice and the wording may differ, and a small tweak to a prompt can quietly make things better in one place and worse in another.
That's why "we tried it a few times and it looked good" is the most common route to an embarrassing launch.
Start with real examples
Before you build much, collect examples of what the feature will actually face. For a support assistant, that means real customer questions, not the tidy ones the team made up. Pull them from emails, chat logs and tickets. Fifty is a decent start, a few hundred is better.
Include the awkward ones: vague questions, spelling mistakes, questions in mixed languages, angry messages, requests outside your scope, and questions where the right answer is "I can't help with that."
Decide what "good" means
Write down the ideal answer, or at least the criteria, for each example. For some tasks, there's a clear right answer, such as the correct category for a ticket or the amount on an invoice. For others, you'll define qualities: accurate, polite, short, uses only approved information, doesn't make promises.
If your team can't agree on what a good answer looks like, the AI can't be judged fairly either. That conversation alone is valuable.
Run it and score it
Run every example through the system and compare with your ideal. For tasks with clear answers, you can score automatically. For open-ended ones, a person reads and rates each result, say pass, partly, or fail, with a short note on why.
Some teams use another AI model to help score answers, which is useful for scale. Treat it as a helper, not a judge. Spot-check its ratings against your own, because it has biases too.
Read the failures
The score is useful. The failures are gold. Read each miss and sort them into causes: it didn't have the information, it misunderstood the question, it ignored an instruction, it made something up, or the source material was wrong. Each cause points to a different fix, and you'll often find the problem isn't the model at all but missing or messy documents.
Change one thing at a time
When you adjust a prompt, switch models or change how documents are retrieved, rerun the whole set. Change one thing per round, so you know what helped. Otherwise you'll end up with a pile of tweaks and no idea which ones matter.
Keep a simple log: what you changed, the score before and after. It saves enormous confusion later.
Test the edges on purpose
Beyond normal questions, try to break it. Ask things it shouldn't answer. Try to talk it into ignoring its instructions, which is called prompt injection. Paste in text that includes hidden commands. Give it conflicting information. Ask for personal data about other customers. These tests show whether your safeguards hold, and they're better found by you than by a stranger on the internet.
Don't skip safety and tone
Quality isn't only accuracy. Check that it stays within your brand voice, avoids promises you can't keep, handles upset people sensibly and escalates to a human when it should. For anything health, legal or financial, check that it doesn't overstep.
Keep checking after launch
Real users will surprise you. Log conversations (respecting privacy rules), give users a simple thumbs up or down, and review a sample every week. Add new failures to your test set so the same mistake can't return unnoticed. Model providers also update models, which can change behaviour, so rerun your tests when anything changes.
A reasonable minimum
You don't need a research lab. A spreadsheet with a hundred real examples, expected answers, a pass/fail column and a weekly review already puts you ahead of most teams. Start there, and grow it as the feature matters more.
