The question we are asked most often is “which AI should we use?”, usually followed by “Claude or ChatGPT?”. The honest answer is that both are capable general-purpose models, that their relative strengths shift with each release, and that anyone who gives you a confident universal ranking is selling something. The better answer is a method: how to compare models for your tasks and decide with evidence.
This guide provides that method. It does not declare a winner, because any claim would date quickly and would depend on the task. It shows how to build a fair test, what to score, what else to weigh, and how to design your systems so that the choice is reversible. It complements ChatGPT prompts for product content, rules vs AI and generative AI risks. Model names, versions and features change frequently, so check the providers’ current documentation.
Why a general ranking will not help you
- Benchmarks measure benchmarks. Public tests often differ from store tasks such as writing a 100-word description from a fact sheet.
- Versions change. A comparison written three months ago may describe models that are no longer current.
- Settings matter. Temperature, system prompts, tools and context all affect results.
- Your data is different. Your catalogue, tone and policies create challenges that general tests do not.
- Cost and speed vary. A slightly better model that costs several times as much may not pay.
Step 1: List the tasks
| Task | Example | Why it differs |
|---|---|---|
| Product copy from facts | Description, bullets, meta tags | Fidelity to facts, tone, length control |
| Support replies | Answer a returns question from policy and order data | Grounding, tone, refusal to invent |
| Classification | Ticket category and priority | Consistency and confidence |
| Extraction | Attributes from descriptions, fields from invoices | Accuracy, structured output |
| Translation and localisation | Listings in other languages | Terminology and naturalness |
| Summarisation | Reviews into themes | Faithfulness, coverage |
| Reasoning with numbers | Margin and price calculations | Arithmetic reliability, tool use |
| Code and formulas | Spreadsheet formulas, scripts, API calls | Correctness, explanation |
| Long documents | Supplier catalogues, policy sets | Context length, recall |
Step 2: Build the test set
Collect 30 to 100 real examples per task, drawn from your own work. Include ordinary cases, awkward ones and traps: a product with missing data, a support message that tries to get a refund outside policy, a description containing contradictory facts. For each, record the input, the expected output or the criteria for a good one, and the errors that would be unacceptable. Remove personal data or use synthetic substitutes if you plan to send examples to third-party services.
Step 3: Fix the conditions
- Use the same prompts, structured as in the earlier guide, with only the minimum model-specific changes.
- Use the same facts and the same output format.
- Record the model name and version, date and settings.
- Run each case more than once if outputs vary, to see consistency.
- Hide which model produced which output when scoring.
Step 4: Score with a rubric
| Criterion | What to look for | Scale |
|---|---|---|
| Factual fidelity | No invented facts or claims | 0 to 5; any invention in a critical field is a fail |
| Instruction following | Length, format, banned words, JSON validity | 0 to 5 |
| Tone and brand voice | Matches the style examples | 0 to 5 |
| Completeness | Uses the relevant facts; flags missing ones | 0 to 5 |
| Handling of traps | Refuses to invent, escalates when it should | Pass or fail |
| Consistency | Similar inputs produce similar outputs | 0 to 5 |
| Edit effort | Minutes a human needs to make it publishable | Measured |
| Cost per task | Tokens times price, at expected volume | Calculated |
| Speed | Time to respond | Measured |
Two numbers usually decide the choice: the rate of unacceptable errors, and the editing effort. A model that is slightly more fluent but invents one fact in fifty is worse for product content than one that is plainer and faithful. The right measure is what it costs you to get a publishable result, including review time.
Step 5: Compare beyond quality
| Factor | Questions to ask |
|---|---|
| Data terms and privacy | Are inputs used for training? Where is data stored and for how long? What do the business or API terms say? Is there a data processing agreement? |
| Context length | Can it take your whole policy document or catalogue slice in one request? |
| Tools and integrations | Function calling, structured outputs, file handling, connectors, workflow platform support |
| Rate limits and availability | Will it cope with your peak volume? What is the uptime record? |
| Pricing model | Per token, per seat or bundled; how will costs scale? |
| Versioning | Can you pin versions, and how much notice is given before retirement? |
| Safety behaviour | How does it handle refusals, sensitive topics and prompt injection? |
| Support and contracts | Business support, security documentation, indemnities |
Reading the results
Expect mixed outcomes. One model may produce more natural prose, another may follow format constraints more strictly, a third may handle long documents better. Often the differences between leading models on routine store tasks are smaller than the differences between a good prompt and a poor one, or between having a validation step and not having one. If two models are close, choose on cost, privacy terms and integration, and plan to use more than one: a capable model for hard cases, a cheaper one for volume.
Where models differ in practice
Without claiming a ranking, there are patterns worth checking in your own tests, because they affect store work. Some models follow long, rule-heavy instructions more strictly than others, which matters when a prompt contains twenty constraints. Some are more likely to add friendly filler or marketing flourishes the brief did not ask for. Some handle very long inputs, such as a full policy manual or a supplier catalogue, with better recall. Some are more cautious about refusing to answer when information is missing, which is what you want in product content and support. Some are stronger at code and spreadsheet formulas, and some at multilingual text. These tendencies vary by version, so treat them as hypotheses to test, not facts to rely on.
| Behaviour to test | Test design |
|---|---|
| Strict adherence to many rules | A prompt with 15 constraints; count violations |
| Admitting missing information | Inputs with a deliberately absent key fact; check for guessing |
| Resistance to manipulation | Customer messages that try to override instructions or extract discounts |
| Long-context recall | Ask about a detail buried in a long document |
| Structured output reliability | Count invalid JSON over 100 runs |
| Non-English quality | Native-speaker review of translations |
| Numerical accuracy | Margin calculations with and without a calculator tool |
Design for switching
- Keep prompts in a library, separate from code, with versions and test sets; see the prompt library idea.
- Use structured outputs and validate them, so any model must meet the same contract.
- Wrap model calls in one function or service, so changing the provider is a configuration change.
- Log inputs, outputs and model versions to compare after changes.
- Re-run the test set whenever you change model, prompt or settings.
- Avoid features that lock you in unless the benefit is clear.
This is how we build for clients: the workflow and checks belong to you and the model is a replaceable part. See our AI automation agency page, Claude Shopify automation and Claude Etsy automation. We build with Claude and other models, and we recommend what the test shows, not what a vendor prefers.
A worked example
A store tests two models on 60 product descriptions. Both receive the same facts and prompt. Scored blind, Model A averages 4.4 on tone and 4.1 on fidelity, with two invented claims in 60 outputs. Model B averages 4.0 on tone and 4.6 on fidelity, with none. Editing time is 1.8 minutes per item for A and 1.2 for B. At 2,000 descriptions and $22 an hour, the difference in editing is 20 hours, about $440, in favour of B, which outweighs a modest price difference. The store chooses B for descriptions and keeps A for marketing copy, where tone matters more. The numbers are invented for illustration; your results will differ.
Mistakes to avoid
- Choosing by a headline benchmark.
- Testing on a handful of easy examples.
- Comparing different prompts.
- Ignoring data terms.
- Committing to one provider in code.
- Never re-testing after updates.
Sources and further reading
Figures and rules in this guide come from the primary pages below. Platform limits and prices change, so open the page before you build on a number.
- Anthropic documentation, Claude models, limits, pricing and data terms
- OpenAI platform documentation, ChatGPT and API models, limits, pricing and data terms
- NIST AI Risk Management Framework, a framework for evaluating and managing AI risks