The API Is a Proxy, Not the Product
If an AI visibility report uses an API to estimate what people see in ChatGPT, the report may still be useful. It is not a faithful recording of the consumer ChatGPT experience.
A new observational preprint compared citations from ChatGPT's web application with Google results and OpenAI Responses API output. For the GPT-5.5 Instant setting in the study, the two external proxies together recovered 26.2% of the exact pages ChatGPT cited and 48.7% of the cited sites. That gap matters when a business is paying for citation tracking, competitor comparisons, or proof that a content change worked.
The practical takeaway is not to throw away every API-based tool. It is to label the data honestly, test the real customer-facing product when possible, and report exact-page matches separately from site-level presence. Otherwise, a tidy dashboard can provide very precise answers to a slightly different question. Marketing technology does enjoy that trick.
What the Researchers Actually Tested
The preprint, "Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations," was posted by researchers from the National University of Singapore and Xi'an Jiaotong University (https://arxiv.org/abs/2610.15382). It studied four ChatGPT web settings across model generations from GPT-5.2 through GPT-5.5, using Instant and Thinking modes.
For each of 80 questions, the researchers repeated ChatGPT runs and observed the production web application's retrieval and citation behavior. They also queried Google and OpenAI's Responses API as external stand-ins. Search Engine Journal summarized the findings and the important limitation that the paper is a preprint, not peer-reviewed research (https://www.searchenginejournal.com/openais-api-only-partly-matched-chatgpts-citations-paper-finds/593304/).
The headline numbers should be read with their boundaries attached:
- The combined Google and API proxies recovered 28.3% of exact pages cited by GPT-5.3 Instant and 26.2% for GPT-5.5 Instant
- At the site level, the combined match rose to 50.3% for GPT-5.3 Instant and 48.7% for GPT-5.5 Instant
- The API's candidate list covered more ChatGPT-cited pages than Google results alone
- The API's own final citations were a closer match than its broader candidate list, but still did not reproduce the web product
- The measurements describe specific models, settings, questions, and collection dates from 2026, not every future ChatGPT experience
Why ChatGPT API Citation Accuracy Has Limits
An API call and a ChatGPT session can share a provider and still run through different serving environments, retrieval steps, query rewrites, model settings, and product logic. The logo on the invoice does not make the pipelines identical.
The paper observed that ChatGPT rewrote questions into search queries rather than simply sending the original wording. It also found meaningful variation between repeated runs. For GPT-5.5 Instant, 44.5% of the distinct pages returned for a question appeared in only one of 20 runs. Site-level overlap was stronger than exact-page overlap.
This creates three separate measurement questions:
Did the system retrieve the site?
Retrieval means a page or site entered the candidate pool. It shows potential availability, not that the source appeared in the answer.
Did the system cite the page?
Citation means the answer exposed a source link. A tool that captures API citations can measure its own API run accurately while still missing links shown in the ChatGPT web application.
Did the system recommend the business?
Recommendation is a different event again. A source can be retrieved without being cited, cited without the brand being named, or used in an answer that recommends somebody else. Counting all three as "visibility" creates a wonderfully smooth chart and a fairly lumpy decision.
What an Owner Should Ask Before Buying Citation Tracking
A business owner does not need a lecture on retrieval architecture. The owner needs to know whether the report is reliable enough to guide spending. Ask the provider these questions before treating the dashboard as evidence.
- What product is being tested? Is the tool recording the consumer ChatGPT interface, an OpenAI API response, Google results, or a model of likely citations?
- Which model and mode are used? Instant, Thinking, search-enabled, logged-in, and other experiences can behave differently.
- Are exact URLs and sites reported separately? A site-level match may reveal authority or entity presence, but it does not prove the intended service page was cited.
- How many times is each prompt repeated? One run can be useful for diagnosis, but repeated runs reveal stability.
- Are prompt rewrites or fan-out queries preserved? These can explain why apparently identical questions reach different sources.
- Does the report save the answer and cited URLs? A percentage without the underlying evidence is difficult to audit.
- How are model changes handled? A baseline built on one deployed setting may need to be reset when the product changes.
- Can results be connected to business outcomes? Citation movement matters more when it affects relevant recommendations, qualified visits, branded demand, leads, or sales.
A provider does not need perfect access to every hidden system. Nobody has that. It should, however, distinguish observed data from inferred data and explain what the method cannot see.
Use API Data Without Pretending It Is ChatGPT
API-based citation tracking can still support useful decisions. It can run repeatable prompt sets at scale, preserve structured outputs, compare competitors, classify sources, and identify pages that appear across many related questions.
Use it as a controlled monitoring layer:
- Keep prompts tied to real buyer questions and commercial categories
- Record the API model, tools, parameters, date, market, and run count
- Separate retrieved candidates, cited pages, cited sites, brand mentions, and recommendations
- Repeat important prompts instead of treating one response as a rank
- Validate a sample in the customer-facing ChatGPT product
- Recheck the method after meaningful model or product changes
- Preserve raw responses and source URLs so a human can inspect surprising results
The goal is not to force API data to impersonate the web application. The goal is to understand what each evidence stream can support.
Report Confidence, Not Just Percentages
A credible AI visibility report should make uncertainty visible. For each prompt cluster, show how many runs were completed, which product was tested, whether the brand was retrieved, cited, mentioned, or recommended, and how stable the result was.
Use confidence labels that describe the evidence:
- Observed in the customer product: Captured directly from the relevant ChatGPT experience
- Observed through the API: Accurate for the recorded API configuration, not assumed to duplicate ChatGPT
- Site-level agreement: The same domain or hostname appeared, but not necessarily the same page
- Exact-page agreement: The same URL appeared in both evidence streams
- Inferred opportunity: A source pattern suggests where to investigate, but no customer-product citation was observed
This prevents a common reporting failure. Suppose an API repeatedly cites a company's research hub while ChatGPT cites different pages from the same domain. The signal may still say the site has topic authority. It does not say the API can predict the exact page a buyer will see.
Spend Based on the Decision the Data Can Support
ChatGPT citation API accuracy is not all-or-nothing. API results can be consistent, scalable, and useful without being a mirror of ChatGPT's consumer product. The new research puts a useful number on that distinction: matching a cited site was much easier than matching the exact cited page in the tested settings.
For business owners, the sensible approach is layered. Use API data to monitor broad patterns. Use direct product checks to validate high-value prompts. Review the actual answer, sources, competitors, and recommendation language. Then connect the findings to content, technical access, third-party authority, qualified traffic, and customer outcomes.
If a report cannot tell you whether its citations came from ChatGPT or from a proxy for ChatGPT, fix the measurement before paying to fix the website. An AI Visibility Audit should reduce uncertainty about what to do next, not decorate it.
