How to Benchmark Competitor Visibility in AI Answers Without Calling It a Ranking

A useful competitor benchmark answers a narrower question than “Who ranks first in AI search?” It asks: In the answers we collected to these reader questions, how often was each publisher named, and how often was one of its pages cited? The prompt set, product and test conditions belong beside every result. Without them, a percentage is difficult to interpret and harder to reproduce.

Define the comparison before collecting answers

Start with publishers a reader might encounter while researching your subject. Include direct alternatives, but consider publishers whose pages appear as sources even when their brands are not named. Ahrefs distinguishes brands mentioned alongside yours, sites cited instead of yours, and brands readers compare with yours. Those are related forms of competition, not interchangeable measurements.

Make a mapping sheet with one row per publisher. Record its primary brand name, accepted name variants and owned domains. Decide whether a product name or sub-brand counts toward the parent before testing. For citations, specify whether subdomains and other owned sites count. Leave ambiguous matches for manual review rather than quietly assigning them to whichever publisher you expected to find.

Then write a fixed library of questions a reader could reasonably ask. Group them by intent, such as informational questions, selection questions and comparisons. Most should be unbranded: “How can a small publisher compare newsletter tools?” tests a different opportunity from “Is Publisher A better than Publisher B?” Keep explicitly branded questions in their own group. Otherwise, a publisher named in the question gets an artificial head start in the reported mention rate.

Save the exact wording and the reason each question belongs in the set. If a prompt is irrelevant, decide that before collecting answers. Changing the library after seeing results makes a later comparison a different test. A fixed query library grouped by intent, repeated measurements and inspection of cited URLs provide a workable basis for comparison.

Run the same prompts under recorded conditions

Choose the AI products you want to examine, then maintain a separate result sheet for each. Record the product, available mode, locale, collection date, account or session conditions, exact prompt and rerun number for every attempt. Save the answer as displayed, including its cited links. If an attempt produces no usable answer, log that outcome rather than deleting the row.

Repeat each prompt under the conditions you have chosen. A second answer is another observation, not independent confirmation of the first. Answer visibility can change with prompt phrasing, location or model, so a single response should not stand in for a stable competitor position. Keep the prompt wording fixed within a benchmark round; if you want to test a wording variation, give it its own prompt ID.

Do not combine products into a headline winner. LLM Pulse’s competitor-benchmarking page notes that a lead in a combined figure can come from one model. Its page describes its own weekly, same-prompt tracking, not a required schedule for every publisher. Report each product first, then examine whether a pattern appears across products. Locale deserves the same care when relevant: a comparison of answers shown under different locations is not a like-for-like rerun.

Score what the answer actually contains

Use two independent yes-or-no fields for each publisher in each usable answer:

• Brand mention: Does the answer text explicitly name the mapped brand or an accepted variant? Do not count a domain that appears only in a citation as a prose mention.
• Owned-page citation: Does a displayed citation link to one of the publisher’s mapped domains? Save the cited URL. Do not infer a citation from a brand mention without a link.

An answer can meet both tests, one test or neither. More than one publisher can appear in the same answer, so publisher rates need not add to 100%. Ahrefs defines mentions as brand appearances in AI responses and citations as appearances of a brand’s website as a source. Your worksheet should state the more precise counting choices you use, including how you handle repeated names or multiple links in one answer.

A practical row layout is: prompt ID, intent, product, mode, locale, date, session condition, rerun number, response status, saved answer, publisher ID, mention flag, citation flag, cited URLs and reviewer note. Have another editor check uncertain aliases and links against the mapping sheet. A citation to an article about a competitor on an unrelated domain is not a citation of that competitor’s owned page under this rule.

Publish rates that can be checked

Here is a hypothetical worksheet calculation, not a measured result. Suppose one product is tested with 12 unbranded prompts, six informational and six comparison prompts, run twice each. That creates 24 attempts. One comparison attempt yields no usable answer.

Intent	Usable answers / attempts	Publisher A mentioned	Publisher A cited
Informational	12 / 12	2 / 12	1 / 12
Comparison	11 / 12	4 / 11	3 / 11
Combined unbranded sample	23 / 24	6 / 23	4 / 23

Under this worksheet’s rules, Publisher A’s mention rate is 6/23, about 26%, and its owned-page citation rate is 4/23, about 17%, among usable answers. The separate 23/24 figure shows answer coverage. The missing response is not silently scored as an absence, and it is not hidden by reporting percentages alone. If branded prompts were also tested, they would get their own table.

Use the saved answers to interpret gaps without assigning a cause you cannot see. A prompt where a rival is mentioned and your publisher is absent is a sensible place to inspect the question, the rival’s description and any cited pages. It does not establish why your publisher was omitted. Likewise, a cited competitor URL identifies a source shown in that answer, not a guaranteed content recipe. LLM Pulse describes prompt-level mention gaps and cited-URL comparisons; those are useful views when their underlying prompts and counting rules remain visible.

For the next round, preserve the original prompt IDs and definitions, and record any additions as a new set. Report changes by product and intent alongside both rounds’ numerators and denominators. The result is a comparison of appearances in sampled answers, not an overall publisher ranking and not a measure of visits or conversions. If you need the latter, use a separate referral and conversion report rather than treating a citation as a recorded visit.