Same Intent, Three Phrasings: Our Pre-Registered Test of Whether Query Wording Changes Who AI Cites

Experiment design — published September 22, 2026, in the GEO Lab. This is a pre-registration: the topic set, the exact phrasings, the metrics, and five falsifiable predictions go out now; we collect and open the data later this week. We’re rooting for the skeptical answer here — that phrasing doesn’t decide citations — which is exactly the answer you have to guard against fooling yourself into. So we planted the goalposts in public before kickoff.

What claim are we testing this week?

The hypothesis inherited from Monday, stated so it can lose: because AI engines expand and rewrite a query before retrieving, a long conversational question pulls in a different — more niche, more long-tail — set of cited sources than the bare head keyword for the same intent. If that’s true, phrasing isn’t just a trigger for whether an answer appears; it’s a lever that decides which kind of source wins the citation, and small specific publishers would gain ground on the long phrasings while broad authority sites dominate the short ones.

The counter-hypothesis, which our own data leans toward: the engine normalizes all three phrasings to roughly the same underlying intent, retrieves from roughly the same candidate pool, and cites roughly the same domains regardless of how the searcher worded it. On that story, “conversational optimization” changes whether the door opens, not who walks through it. Both are plausible; the whole GEO industry has been giving advice as if the first were settled. This experiment is built to make the data pick a side.

What exactly gets collected — the topics and the three phrasings?

Eight commercial, buyer-intent topics — the queries where recommendations and citations actually carry money — and for each one, three phrasings that hold the intent fixed while stretching the wording from a bare keyword to a full natural-language question:

We run all 24 phrasings on ChatGPT (web search) and Google AI Mode — the same two engines the rest of this series uses, so this extends the dataset rather than starting a new one. Collection is logged-out, a single snapshot, excluding each engine’s self-citations and Google’s own properties (the exclusions we’ve held all series). Perplexity sits out again: it rate-limits hard mid-collection, and a half-populated third engine would add noise to a test whose whole point is measuring small changes precisely.

How do we tell a real phrasing effect from random wobble?

This is the part most “we tested it” posts skip, and it’s where the honesty lives. AI citation lists are not deterministic — ask ChatGPT the identical question twice and the set of cited domains often isn’t identical. So if the domains change between a short keyword and a long question, some of that change is the phrasing and some is just run-to-run randomness. You can’t attribute the difference to phrasing until you know how big the randomness alone is.

So we build a control. For every topic and engine, we re-run the medium phrasing a second time, unchanged, and measure how much the cited-domain set moves between two identical asks. That’s the noise floor. Then the phrasing effect is only real if the cross-phrasing sets move more than that floor. We measure “how much the set moved” with the Jaccard overlap — the size of the shared domains divided by the size of the combined set, where 1.0 means the two answers cited exactly the same domains and 0.0 means no domain in common. Three numbers do the work:

• Noise-floor overlap — Jaccard between the two identical medium-phrasing runs. How stable citations are when nothing changes.
• Cross-phrasing overlap — mean Jaccard across the three phrasing pairs (short↔medium, medium↔long, short↔long) for the same intent and engine. How stable citations are when only the wording changes.
• Authority share — the fraction of cited domains that sit in the high brand-demand tier, computed per phrasing with the same concentration math as our 15-domains study. Tells us whether long phrasings tilt toward niche sources.

If cross-phrasing overlap sits right around the noise floor, phrasing did nothing selection can’t be blamed on chance. If it collapses far below the floor — and long phrasings systematically drop authority share — phrasing is a genuine lever and our prior is wrong. Everything is reported per engine; there is no blended average, because the two engines retrieve so differently that averaging them hides the answer.

What five predictions are we locking before collection?

Pre-registered, numbered, each falsifiable. If the data contradicts one, Thursday’s post says so in plain language — no quiet reframing.

1. P1 — Phrasing barely beats noise. Cross-phrasing overlap is within 0.15 Jaccard of the same-phrasing noise floor on both engines. In plain terms: changing the wording moves the cited domains about as much as re-asking the identical question does. This is the load-bearing prediction — if cross-phrasing overlap collapses well below the noise floor, phrasing is a selection lever and Monday’s skeptical prior is wrong.
2. P2 — Long phrasing does not hand citations to niche sites. Authority share drops by less than 15 percentage points from the short to the long phrasing (averaged across topics, per engine). The niche-shift hypothesis predicts a large drop as small publishers win the long-tail; we predict it stays roughly flat.
3. P3 — Trigger moves, selection doesn’t. Long conversational phrasings produce a citation panel in a higher share of runs than short keywords — the well-supported half of the trend — but among topics where all three phrasings fire, cross-phrasing overlap still sits near the noise floor. The effect is on whether an answer appears, not on who’s cited inside it.
4. P4 — Whatever phrasing does, it’s engine-dependent. Google AI Mode shows higher cross-phrasing overlap than ChatGPT — it’s grounded on Google’s ranked index, so it stays closer to the same domains, while ChatGPT’s own retrieval and query expansion wobble more. We report the two engines separately and expect the gap between them to exceed the gap phrasing itself creates.
5. P5 — There’s a phrasing-invariant core. For at least 60% of topics on at least one engine, one or more domains are cited under all three phrasings. A stable backbone survives the rewording, consistent with a fragmented citation economy where a few domains anchor each intent (top 15 = 23%) and everything else is churn.

Together these describe one shape: phrasing is a doorbell, not a doorman. It changes how often the AI answers, not which sources it reaches for once it does. That’s the opposite of the “rewrite everything for conversational queries and watch your citations climb” narrative. It also lines up with what we already found moves selection — earned brand mentions correlating at 0.664 and third-party mentions beating owned content — none of which is about how the searcher typed the query. If P1 fails, we were wrong, and Friday’s verdict will say it out loud.

What could make this experiment misleading — and how do we stay honest?

Four limits, named in advance. One: one noise-floor sample is a floor, not a full distribution. A single identical-repeat per topic estimates run-to-run wobble roughly; if we had unlimited quota we’d repeat every phrasing many times. We’ll treat the noise floor as an approximate band, not a hard line, and flag any result that hinges on a close call. Two: the phrasing tiers involve judgment. “Short,” “medium,” and “long” are our calls, so we publish the exact query strings and invite re-runs — someone else’s phrasings might behave differently, and that’s a fair critique. Three: the authority-tier proxy is imperfect — a public traffic/demand estimate is a stand-in for true brand demand, and we’ll disclose borderline domains. Four: it’s one logged-out snapshot of 8 topics on two engines — a strong signal in one place and time, not a universal law, and citation sets drift and vary by engine.

We design in the open — same as the rest of the GEO Lab — because the answer we expect is the convenient one for us: we’ve told readers to fix citations with answer-first structure and earned mentions, not query cosmetics, and “phrasing doesn’t matter for selection” flatters that advice. The noise floor and the locked predictions exist so the data can still embarrass us. Predictions are set. Wednesday we collect; Thursday we open the full per-topic overlap sheet; Friday we judge all five against exactly what’s written here.

Frequently asked questions

What is this experiment actually testing?

Whether changing only the phrasing of a search — from a short keyword to a long conversational question, with the intent held fixed — changes which domains an AI engine cites. It’s the measured follow-up to Monday’s argument that the “write for conversational queries” trend moves whether an AI answer triggers, but has never been shown to move which sources get cited once it does.

Why re-run the same phrasing twice as a control?

Because AI citation lists aren’t deterministic — ask the identical question twice and the cited domains often differ. Without measuring that run-to-run wobble, you’d mistake random churn for a phrasing effect. The same-phrasing repeat gives us a noise floor: a phrasing effect only counts if changing the words moves the cited-domain set more than re-asking the same words does.

Why only 8 topics and two engines?

Depth over breadth. Testing three phrasings plus a repeat control across two engines is 32 collections per topic-set, and precision matters more than volume when the effect you’re measuring is small. Eight buyer-intent topics span shopping and software where citations carry commercial weight. ChatGPT and Google AI Mode are the two engines the rest of this series uses; Perplexity sits out because it rate-limits mid-collection and would add noise to a test built to measure small changes.

What result would prove the “conversational optimization” advice right?

If cross-phrasing overlap collapses well below the same-phrasing noise floor — meaning the long question cites materially different domains than the keyword — and long phrasings systematically drop authority share toward niche publishers, that’s phrasing acting as a real selection lever, and we’d be wrong. Our locked prediction (P1) is the opposite: phrasing barely beats noise. Either way the pre-registered record decides it, not a post-hoc story.

When are the results published?

We collect Wednesday and publish results Thursday with the full per-topic overlap sheet and charts, then a verdict Friday that judges each of the five predictions and turns the finding into an action list. This post is the locked design; nothing here changes once the data is in.

Sources

• Does How You Phrase a Search Change Who AI Cites? — this week’s Monday fact-check that split trigger from selection and set up the hypothesis this experiment tests.
• Cross-engine citation results — 2 of 395 shared domains; why any phrasing effect rides on top of engines that already diverge.
• Rank vs AI citation overlap — 34.4% blended, evidence that selection isn’t governed by the obvious inputs.
• Do 15 domains control AI citations? — the concentration math behind the authority-share metric and P5.
• Brand mentions beat backlinks and earned beats owned — what does move selection, none of it about query wording.
• GeoParrot GEO Lab — pre-registered design; collection to follow across ChatGPT (web search) and Google AI Mode, logged-out, single snapshot, 8 buyer-intent topics × 3 phrasings + same-phrasing repeat control.

GeoParrot is a GEO Lab: we test what AI search actually cites and recommends, then publish the method and the misses. This is a Tuesday pre-registration in a four-part weekly arc — what AI SEO is, measured rather than asserted. Monday set the hypothesis; Wednesday we collect; Thursday the data; Friday the verdict.