"Which AI model should we use?" is one of the first questions clients ask when a project includes AI. The honest answer is that it depends on the task, and that the best way to find out is to test. New models arrive every few months, prices change and a model that leads public rankings is not always the best choice for a specific business task.
Here is the process we follow on client projects. It is the same approach described in our guide on how to choose an AI model for your business, with notes on how it plays out in practice.
Step 1: Define the task precisely
"Use AI for customer support" is not a task. "Read an incoming support email, classify it into one of eight categories and draft a reply for routine categories using our help articles" is. The precise version tells us what the model must read, what it must produce, how long inputs are and what a good result looks like.
We also write down the constraints: how fast it must respond, how many requests a day, what happens when it is wrong and who checks the output.
Step 2: Check the data rules
Before looking at quality, we check what data the model will see and what rules apply. Personal data, health information, financial records and client contracts may limit which services can be used, under which terms and in which regions. Some clients require that data never leaves their own servers, which points to an open-weights model hosted privately.
This step often narrows the list more than any benchmark. See how to keep customer data safe when using AI.
Step 3: Build a short list
We usually test three to five candidates, chosen to cover different trade-offs:
- A frontier model from a major provider, to see the best achievable quality.
- A smaller, faster, cheaper model from the same or another provider.
- Where data rules or cost suggest it, an open-weights model.
- For specialized tasks, a model built for them, such as an embedding model for search or a speech model for transcription.
Our AI model directory helps with this step. Each entry lists the provider, strengths, limits and context window, checked against official documentation and dated.
Step 4: Build a test set from real examples
This is the most important step. We collect thirty to a hundred real examples of the task from the client, including awkward ones: long emails, unclear requests, documents in poor condition, questions the system should refuse. For each, we write down what a good answer looks like, with the client's help.
Public benchmarks measure general ability. A test set of your own examples measures what you actually need.
Step 5: Run the candidates
We run every candidate on the test set with the same instructions, then score the results. Depending on the task, scoring is automatic (did it pick the right category, is the output valid structured data) or done by people (is this reply accurate, helpful and in the right tone). Often it is both.
We record accuracy, the kinds of mistakes each model makes, speed and cost per request. The kinds of mistakes matter as much as the count. A model that says "I am not sure" when uncertain can be better than a slightly more accurate model that guesses confidently.
Step 6: Compare cost at real volume
A model that is twice as good but twenty times more expensive may still be the right choice for a low-volume, high-value task. For a high-volume task, it rarely is. We estimate monthly cost at expected volume for each candidate, including longer inputs such as documents.
We also look at combinations: a cheap model for most requests with a stronger model for difficult ones, or rules that handle clear cases before AI is called at all.
Step 7: Decide, and write down why
We recommend a model with the evidence: test results, costs and trade-offs. The client decides. We write down the reasons, because the decision will be revisited, and the next person should know what was tested and why.
Step 8: Plan for change
The model you choose today will not be the best option in a year. So we design for switching:
- The model is a setting, not something woven through the code.
- Instructions and test sets are kept in version control.
- When a new model arrives, we rerun the test set and compare.
- We watch provider announcements for deprecations and price changes.
This turns model upgrades into a routine check instead of a project.
Mistakes we see
- Choosing the most famous model without testing.
- Testing on invented examples instead of real ones.
- Comparing accuracy but ignoring cost at volume.
- Ignoring data rules until late in the project.
- Hard-coding one model so switching becomes expensive.
An example of the trade-offs
Consider a feature that classifies incoming documents and extracts a handful of fields. On a test set of real documents, a large frontier model and a smaller model from the same family might score almost the same on clean documents, with the larger model doing better on poor scans. At the expected volume, the smaller model could cost a fraction as much. A sensible design uses the smaller model by default and sends low-confidence documents to the larger one or to a person. This is illustrative, but it is the shape of many real decisions.
Read more
To compare models side by side, try our AI model compare tool. For the question of whether you need to train a model at all, see RAG vs fine-tuning. And if you want help evaluating models for your own use case, see our AI model evaluation service or get in touch.
