Quick answer: AI visibility platform Envoyra ran the same 170 buyer-style questions through ChatGPT (GPT-4o), Claude (Sonnet 4.6), Gemini (2.5 Flash), and Perplexity (sonar-pro) once a week for four weeks (Sept. 13 – Oct. 4, 2026) across five B2B SaaS categories. Of 680 question-and-assistant combinations, 536 — 79% — recommended a different top brand at least once across the four runs; only 144 stayed identical every week. The detail that matters for reading the number correctly: ChatGPT and Perplexity had live web search on, but Claude and Gemini didn’t. That means some of this “instability” is almost certainly fresh search results changing week to week — and some of it is two models with no web access giving different answers to the same question anyway, which is a different problem with a different fix.
Trend watch — published October 23, 2026. The study, run by AI visibility platform Envoyra, was covered by Search Engine Land on October 22. Envoyra has not published its test questions, and results reflect specific API models and settings — GPT-4o, Sonnet 4.6, Gemini 2.5 Flash, sonar-pro — which may behave differently from the consumer-facing chat apps most buyers actually use.
What did Envoyra actually test?
170 buyer-style questions per assistant, run once a week for four weeks, across five software categories: CRM, customer support, meeting assistants, marketing automation, and a fifth category not detailed in coverage. That’s 170 × 4 assistants × 4 weeks = 2,720 AI-generated answers, collapsing to 680 unique question-and-assistant pairs once you set aside the week-over-week repeats. The assistants split cleanly into two pairs on one axis that the top-line number glosses over: ChatGPT (GPT-4o) and Perplexity (sonar-pro) both had live web search enabled for the test; Claude (Sonnet 4.6) and Gemini (2.5 Flash) didn’t. The four assistants agreed on the top brand in only one of five categories — customer support, where Zendesk ranked first across all four every time, and Intercom still showed up in 62 of 96 customer-support answers across the four weeks despite not winning the top slot. CRM leaned toward consensus without reaching it: HubSpot led in three of four assistants (appearing in 305 of 368 eligible CRM answers), while Gemini alone favored Zoho CRM. Meeting assistants and marketing automation split further — ChatGPT picked Otter.ai for meetings, Claude and Perplexity both picked Fireflies.ai, Gemini picked Zoom; for marketing automation, ChatGPT named HubSpot, Claude named Klaviyo, and both Gemini and Perplexity named ActiveCampaign.
Is 79% one instability number, or two different phenomena stapled together?
This is the part the coverage doesn’t separate out, and it’s the reason the headline number is less useful to a GEO practitioner than it looks. For ChatGPT and Perplexity, week-to-week change in a recommended brand is mechanically unsurprising — they’re querying live search results, and search results, rankings, and recent news about a vendor can genuinely shift inside a month. A changed answer there could reflect a real, trackable event: a competitor shipped a feature, got a review surge, or had a bad week in the press. But Claude (Sonnet 4.6) and Gemini (2.5 Flash) were tested without web search, running on the same frozen model weights all four weeks, against (presumably) the same or near-identical question text. If their answers still changed, that’s not the world changing — it’s sampling variance, token-level randomness, or subtle differences in how the question was phrased between runs, unless Envoyra fixed temperature to zero and logged exact prompts (neither is confirmed in the coverage, and the firm hasn’t published its question set). That’s a materially different problem for a brand trying to act on this: you can build visibility signals to influence what a live-search assistant finds this week, but you can’t “SEO” your way around a non-browsing model rolling different dice on the same prompt. The study reports one number for both mechanisms. We’d want the split before trusting either half of it as actionable.
| What’s confirmed | What’s not separated out |
|---|---|
| 79% of 680 question-and-assistant pairs changed their top recommended brand at least once across 4 weekly tests (Sept. 13–Oct. 4, 2026) | How much of that 79% comes from live web-search assistants (ChatGPT, Perplexity) vs. non-browsing assistants (Claude, Gemini) running the identical model on the same question |
| 4 assistants agreed on the top brand in only 1 of 5 categories tested (customer support — Zendesk) | Whether Claude/Gemini’s instability reflects sampling temperature, prompt-phrasing drift between weekly runs, or a genuine answer-generation effect — Envoyra hasn’t published its exact test questions or model settings |
| CRM, meeting-assistant, and marketing-automation picks diverged by assistant, not just by week | Whether results generalize to the consumer chat apps most buyers actually use — the test ran against specific API models (GPT-4o, Sonnet 4.6, Gemini 2.5 Flash, sonar-pro), not necessarily what’s live in the ChatGPT or Gemini apps today |
How this connects to what we’ve already measured
We’ve run two different cuts of this same general problem, and this study sits at the intersection of both without fully matching either. In October, we measured same-day disagreement across engines directly: a vendor claimed a 27% cross-engine citation disagreement rate, and our own count on the same topics, same day, put it closer to 99% — a gap we traced to how narrowly “disagreement” gets defined, not to the engines actually agreeing more than we found. A separate, much larger 116,000-citation study later landed closer to our number than the vendor’s, which is the closest thing we have to independent confirmation that cross-engine fragmentation is real and large, not a measurement artifact of our small sample. Both of those were same-day, cross-engine snapshots. What we hadn’t directly measured before is the axis Envoyra is pointing at here: the same engine, asked the same thing, a week apart. The closest prior data point we have is single-engine, not cross-engine — ChatGPT cut its YouTube citation share by 91% in a single month, which we used to argue that chasing any one engine’s monthly source mix is a losing game because the baseline itself won’t hold still. Envoyra’s finding is the brand-recommendation version of that same instability, measured across four engines instead of one. Put together, the honest picture is: engines disagree with each other on any given day (our 99% number), individual engines shift within themselves over weeks to months (our ChatGPT source-mix finding), and now there’s early evidence that happens over four-week windows too, for at least some of the mechanism.
A practitioner checklist for this kind of instability
- Don’t treat one favorable AI answer as a stable win. If 79% of question-assistant pairs changed at least once in four weeks, a single “we’re the #1 recommended tool on ChatGPT” screenshot is a snapshot, not a status.
- Separate your tracking by whether the assistant uses live search. If you’re monitoring AI visibility, split ChatGPT/Perplexity-style (web-search-backed) results from Claude/Gemini-API-style (frozen-model) results — they respond to different inputs and need different interventions.
- For web-search assistants, the lever is the same one GEO always points to: recent reviews, fresh content, and third-party coverage are more plausibly driving week-to-week swings there than for non-browsing models.
- For non-browsing assistants, don’t assume a content play moves the needle quickly. A model’s answer to a fixed prompt, running on frozen weights, changes on a schedule set by the vendor’s next training or fine-tuning cycle — not by anything you publish this week.
- Ask any AI-visibility vendor selling “track your AI rank” dashboards whether they control for sampling temperature and fixed prompts. If they re-run the same question and get a different answer with no web access and no reported randomness control, the dashboard is measuring noise, not your visibility.
The honest caveats
Envoyra hasn’t published its test questions, so we can’t verify whether phrasing was held exactly constant week to week or whether sampling temperature was fixed and reported — both of which would materially change how much of the 79% is “noise” versus “signal.” The study used specific API model versions (GPT-4o, Sonnet 4.6, Gemini 2.5 Flash, sonar-pro) rather than the consumer chat apps, and API defaults don’t always match what ships in the product most buyers use. Five categories and four weeks is a real test, but it’s still a narrow slice of B2B SaaS, not a general claim about all AI-assisted purchase decisions. And our own supporting data points — the 99% cross-engine disagreement count and the ChatGPT source-mix swing — are both from different time windows and different question sets than Envoyra’s; we’re drawing a connection between three independently-measured instability findings, not re-deriving Envoyra’s number from our own data.
Frequently asked questions
What did the Envoyra study actually find?
Testing 170 buyer-style questions across ChatGPT, Claude, Gemini, and Perplexity once a week for four weeks (Sept. 13–Oct. 4, 2026) in five B2B SaaS categories, Envoyra found that 536 of 680 question-and-assistant combinations (79%) recommended a different top brand at least once across the four weekly runs. The four assistants agreed on a category’s top brand in only one of five categories tested.
Why does it matter that Claude and Gemini didn’t have web search on?
ChatGPT and Perplexity were tested with live web search, so their answers changing week to week could reflect real changes in search results or recent coverage of a vendor. Claude and Gemini were tested without web access, running the same frozen model weights each week — so if their answers to the same question still changed, that’s more likely sampling randomness or prompt-phrasing drift than a reflection of anything happening in the real world. The study’s single 79% figure doesn’t separate these two mechanisms.
Does this match GeoParrot’s own data?
Partially, and on a different axis. We’ve measured same-day cross-engine disagreement directly (our count landed near 99% against a vendor’s 27% claim, later supported by an independent 116,000-citation study) and single-engine volatility over time (ChatGPT cut its YouTube citation share by 91% in one month). Envoyra’s week-over-week brand-recommendation instability is a third, related finding we haven’t independently tested ourselves, but it’s consistent with the general pattern: nothing about AI search output — which engine, which week, which source mix — holds still for long.
Should I stop tracking AI brand recommendations if they’re this unstable?
No — but track it as a trend across many weeks, not a single snapshot, and split your tracking by whether the assistant has live web access. A one-time “we’re recommended by ChatGPT” result tells you almost nothing on its own; a pattern held across several weeks, on a web-search-backed assistant, is more likely to reflect something you can actually influence with content and third-party coverage.
Is this the same thing as cross-engine disagreement?
No, they’re related but distinct. Cross-engine disagreement (our 99% figure) measures whether different AI engines agree with each other on the same day. Envoyra’s study measures whether the same engine agrees with itself a week later. A brand could theoretically have high cross-engine disagreement and low week-to-week volatility, or vice versa — the two numbers answer different questions about the same underlying unreliability.
GeoParrot is a GEO Lab: we run experiments to test what AI search engines actually reward, and we grade the industry’s claims — including our own — against what was actually measured. See the running scoreboard at our 2026 GEO benchmark. Related: our same-day cross-engine disagreement count, the 116,000-citation study that landed closer to our number, and why chasing any one engine’s monthly source mix is a losing game.
Leave a Reply