One English Test Can Hide a Customer Gap
Multilingual AI recommendations can differ even when the buyer asks the same question about the same category. If your business serves customers in more than one language, an English-only visibility check may show a reassuring answer while another language recommends different competitors. The practical response is not panic or a rush to translate every page. It is a controlled test across the languages, models, buyer questions, and markets that can actually produce revenue.
What the New Research Actually Found
Search Engine Journal published an exploratory study using 1,650 AI responses across shopping and booking scenarios, five languages, and several models (https://www.searchenginejournal.com/lost-in-translation-how-language-changes-which-brand-ai-recommends/591585/). The core Claude test used English, Arabic, Russian, Simplified Chinese, and Persian. In some categories, the leading recommendation stayed fairly stable. In others, language coincided with large changes in which brand won.
The airline example was especially sharp. In the reported Claude runs, British Airways was selected in 52.5% of Persian responses and none of the Russian responses. EasyJet moved in the other direction, from 12.5% in Persian to 72.5% in Russian. Same scenario, same model, same collection period, different language.
Those numbers are useful as a warning, not a universal benchmark. The research covered a limited set of categories, prompts, models, and one collection period. Some web-enabled responses were truncated, and forty runs per language-category combination cannot describe the whole market. It is a reason to check your own situation properly.
Anthropic's separate research on nearly 310,000 conversations found that the values expressed by Claude varied across languages and models (https://www.anthropic.com/research/claude-values-models-languages). That study did not test brand recommendations, but it supports the broader point that changing language can change more than the vocabulary of an answer.
Decide Whether This Risk Applies to Your Business
Run a multilingual check when one or more of these conditions are true:
- Sales, booking, or support data shows customers using more than one language
- The business targets countries or neighborhoods with distinct language audiences
- Translated or localized pages already attract meaningful traffic or leads
- Customer reviews and sales conversations regularly switch languages
- Distributors, franchisees, or local teams use different category terms
- A missed recommendation could affect a valuable booking, contract, or recurring customer
Start with commercial evidence, not a map of every language spoken on Earth. If Spanish-speaking customers produce 18% of qualified inquiries, Spanish belongs in the test. If a language has no audience, offer, service capacity, or conversion path behind it, adding it to a dashboard will create activity without a decision. Marketing already has enough decorative exercise.
Build a Multilingual AI Recommendation Test
A useful test keeps the buying intent stable while changing one variable at a time. Otherwise, you will not know whether language, location, model, live web access, or prompt wording caused the difference.
1. Choose questions tied to revenue
Use five to ten questions customers ask before they choose. Examples include “Which accounting firm is best for a bilingual small business?” or “What are three reliable hotels near the convention center?” Pull the questions from calls, forms, reviews, sales notes, and search data. Avoid prompts invented only because they sound impressive in a report.
2. Translate intent, not merely words
Have a fluent speaker or qualified translator adapt each question for the market. Literal translation can preserve grammar while losing the way a real buyer describes the problem. Keep the service need, budget, geography, and decision criteria comparable, then document any cultural or commercial adjustment.
3. Hold the test conditions steady
Use the same model version, account state, location assumptions, web-search setting, prompt structure, and collection window. Run each question several times because AI answers vary. Record the full response, recommendation order, citations, dates, and whether the model searched the web.
4. Separate four different outcomes
Track whether the business is:
- Mentioned: the brand appears somewhere in the answer
- Recommended: the brand is presented as a suitable choice
- Cited: the answer links to the business or a source discussing it
- Accurately described: services, location, pricing context, and fit are correct
A brand can be mentioned but not recommended, recommended with incorrect details, or omitted while a competitor earns the citation. Combining those outcomes into one “visibility score” makes the report tidier and the decision worse.
Compare Gaps by Language, Not Just Overall Score
Create one view per language and buyer question. Note which competitors appear, which sources support the answer, and which claims or category terms repeat. Then compare the patterns.
Look for gaps that can change customer acquisition:
- Your business wins in English but disappears in another high-value language
- The brand appears, but the answer describes the wrong service or market
- Competitors have credible local-language reviews, guides, or directory coverage that you lack
- The model cites third-party sources with outdated or conflicting information
- A translated page exists, but it does not answer the buying question or provide local proof
- The website attracts multilingual traffic but sends everyone into an English-only booking or contact path
Not every difference is a defect. A competitor may genuinely fit one market better because of support, regulation, location, or price.
Fix the Evidence Behind the Answer
If testing reveals a valuable gap, start with the public information a customer and a retrieval system can verify. Build or improve localized pages that explain the offer, geography, eligibility, costs, limitations, and next step in the language the customer uses. Add market-relevant proof such as local case studies, accurate reviews, credentials, policies, and third-party coverage.
Google recommends using different URLs for different language versions and using hreflang to help Search understand localized variants (https://developers.google.com/search/docs/specialty/international/managing-multi-regional-sites). It also recommends making the page language obvious and avoiding automatic redirects based only on a guessed language. Those are search guidelines, not a guarantee of an AI recommendation, but they help make each version accessible and understandable.
Keep important business facts aligned across languages. The service name can be localized; the address, operating hours, coverage, credentials, price basis, and cancellation policy should not develop separate personalities. Translation without governance is how one company becomes three slightly different companies online.
Measure Business Impact Before Scaling
Retest the same question set after meaningful changes, not after adjusting one adjective. Compare recommendation frequency, accuracy, citations, and source mix over several collection periods. Then connect the findings to business data by language: qualified inquiries, bookings, close rate, revenue, cancellations, and support burden.
Suppose a hotel receives 400 monthly visits to its Spanish pages but only eight booking starts, while English pages convert 1,600 visits into 96 starts. The start rates are 2% and 6%. That gap does not prove AI caused the problem, but it gives the owner a commercial reason to inspect recommendation accuracy, page quality, offer fit, and the booking path. The language test helps locate a possible leak; customer and revenue data decide whether fixing it deserves budget.
Avoid the Expensive Shortcuts
Common mistakes include:
- Testing one prompt once and calling the result a market trend
- Using machine translation without a fluent-market review
- Comparing different models, dates, and web settings as if language were the only variable
- Publishing thin translated pages with no local proof or useful next step
- Measuring mentions while ignoring recommendations, accuracy, and customers
- Assuming every language gap can be fixed through website copy alone
- Promising that localization will force AI systems to recommend the brand
Multilingual AI recommendations are variable, and the available research is still early. That is precisely why the test should be bounded, repeatable, and connected to decisions. Precision beats a giant monitoring spreadsheet nobody uses.
Test the Languages Your Customers Actually Use
An English result cannot stand in for every customer. If a meaningful audience buys in another language, test the same high-value questions in that language, preserve the conditions, repeat the runs, and compare the evidence behind the answers.
Then fix what is real: missing localized information, weak proof, inconsistent facts, inaccessible pages, or a conversion path that stops at the language switch. An AI Visibility Audit can include this multilingual layer when the business case supports it. The outcome is not a promise that every model will choose you. It is a clearer view of where qualified customers may be losing confidence or choosing someone else.
