Why AI Recommendations Change and How to Measure Them

Ask an AI assistant the same question twice and you may get two different lists. Ask from a different place and the lists change again. So one screenshot, yours or a salesperson's, tells you almost nothing. A real check asks the way visitors ask, from places visitors stand, many times, and writes down every answer. This page covers that measurement; our guide to how AI assistants choose local businesses covers what shapes the answers.

Ask twice, get two answers

BrightLocal, a company that sells local search tools, published a study on September 16, 2026. It ran about 200,000 non-branded prompts for more than 1,300 business locations over 60 days, with about three report runs per location. The study does not name a country; its examples are American. When a search was repeated, the businesses named overlapped only 21 to 33 percent: 23 percent for ChatGPT, 21 percent for Google's AI Mode and 33 percent for AI Overviews. A business named once showed up again in about half of the later runs.

The same study compared surfaces. Tracked businesses appeared on Google Maps in 66 percent of searches, against 32 to 38 percent on the AI surfaces. ChatGPT's lists shared only 19 to 20 percent of their businesses with either Google AI surface. So the assistants disagree with each other, not only with themselves from one run to the next.

This is data from a vendor with a tracking product, and the study notes that longer lists lower overlap scores, which can make answers look less steady than they are. We have not yet published our own Seoul repeat-run counts, so do not assume Seoul looks the same.

Location changes the answer

OpenAI says ChatGPT search may estimate a general location from your IP address and may share that general location with its search providers to make results more accurate. Its example: ask for restaurants near you from the San Francisco area, and ChatGPT may search for "top restaurants San Francisco." Sharing your device location is optional and off by default. So by default, ChatGPT likely places you roughly, by network, not by the street you stand on.

Print of a street map with an ochre shop pin among cream Korean rooftops, where three small black figures (one a block away, one on top of a tall teal hotel tower, one far off beside a paper plane) each hold up a speech bubble listing rose, ochre and teal dots in a different order, with dashed lines running from each figure to the pin.

Google's side varies too. Whitespark's May 22, 2026 guide to AI Mode describes two people typing the same query from the same place and getting very different responses. It calls AI Mode a black box and advises trying your customers' questions in it and noting which sources it cites for you and your competitors.

Distance matters inside one town. BrightLocal calls proximity "still a key factor" in local AI answers: about half or more of the businesses named were within 5 km. Answers from different points in the same town also differ, overlapping 36 percent for ChatGPT and about 47 percent for the two Google surfaces. Local Falcon, another tracking vendor, ran every search in its hotel study from nine grid points within a two-mile radius of each hotel.

In Seoul, your counter in Mapo, a hotel lobby in Myeongdong and the arrivals hall at Incheon are three different starting points, and your visitor is at the second or third. Some visitors ask before they fly, from Tokyo or Sydney. By default ChatGPT then places them by their home network, so "near me" points at home. Expect them to name the neighborhood instead: "best Korean BBQ in Mapo." Our guide on why your Maps rank changes every few blocks shows the same effect on Google Maps.

What Search Console shows about AI answers

Google announced Search Console's generative AI performance reports on June 3, 2026. Per its launch post and help page, the report shows impressions only: how often links to your site appeared in AI Overviews and AI Mode on Google Search. Discover gets a report of its own. You can split impressions by page, country, device and date. The country split helps here: it shows whether your pages appear for people searching in Korea or abroad. Neither page mentions clicks or search queries, so you cannot see which question triggered an appearance. If two of your results show in one AI answer, the chart counts one impression for your site. Search Labs experiments are excluded. Nor does either page cover Google Maps, your Business Profile or ChatGPT.

On rollout, the launch post says Google began with a subset of sites. A note on the help page now says that as of August 31, 2026, the insights reached all websites worldwide. The report may still not appear if your site has had too few impressions in these features, or if you have excluded your site from them. And Search Console only collects data for a site once someone adds it, so if your site is not there yet, add it now. If your shop has no website, this report has nothing to show, and the manual check below matters more.

Run your own check

Write ten questions the way a visitor would type them. Say you run a hanbok rental near Gyeongbokgung. Your list might include "Where can I rent a hanbok near Gyeongbokgung in English?", "Best hanbok rental for a family with kids in Seoul" and "Hanbok rental open late near Anguk Station". If you run a restaurant, try "vegetarian Korean restaurant in Mapo" or "Korean BBQ open on Chuseok near Hongdae". Visitors ask for something specific: English staff, walk-ins, vegetarian, open on a holiday. Include a few questions that name your neighborhood, the way someone planning from home would ask.

Run each question several times, in a fresh chat each time with ChatGPT's memory turned off. OpenAI says ChatGPT may use saved memories when it turns your question into a search. Run the questions from at least two places: your shop, and somewhere a visitor stands, such as a hotel lobby or a cafe near a station on mobile data. ChatGPT places you by network unless you share device location, so two spots in one city may look the same to it; there, asking with and without your neighborhood's name shows more. Do it in each assistant you care about, starting with ChatGPT and Google's AI Mode. Ten questions, three runs, two places and one assistant is 60 answers. It takes a long afternoon, so do it once, then repeat monthly.

Log every answer, including the ones that name you and the ones that do not. Keep the exact wording of the question. A tally of only the good results is not a measurement.

What to record

For every answer, write down the date, the assistant, the place and the exact question. Then record four things.

Mentioned or not. Yes or no is enough.
Position. First, third, or in the list at all.
How you are described. Copy the words. Look for wrong hours, a wrong neighborhood, "closed" when you are open, or praise that matches what your customers actually say.
Which sites are cited. Note every source the answer links or names, yours and others'. Our guide to where a Seoul business should be listed for tourists explains which of those sites to care about.

Also record who else gets named. If the same three competitors appear in most answers, that is the list you are trying to join.

Compare with your Maps rank

AI visibility and Maps rank are different measures, and doing well on one does not guarantee the other. Local Falcon's Hotel AI Visibility Index, published September 17, 2026, covered 998 US hotels. In 620 of the 859 searches where a hotel did not appear on Google Maps, at least one chatbot named it, which is 72.2 percent. "Named" meant named at one or more of the nine grid points, a lenient bar. Yet BrightLocal found Maps shows tracked businesses far more often than AI does. Both can be true: AI answers are built differently and often name different businesses, and Maps is still the larger door.

Print of two cream hanok gates, a wide one framing a 10 by 10 dot grid with 66 solid rose dots under a rose map pin and a narrow one framing a grid with 35 solid indigo dots under an indigo chat bubble, an ochre star between them, a check mark at the lower left and a "?" at the lower right.

So track both. Check your rank on Maps from the same places you ask the assistants, and keep the two records side by side. If you rank high on Maps but never appear in AI answers, that is a different problem from the reverse. See the Maps ranking guide for how to read the Maps side.

Read the results honestly

Vendor studies describe what happened, not why, and both studies above come from companies that sell tracking. BrightLocal's own page even gives two different totals for its prompt count. Use them to learn that answers vary, not to learn what moves your result.

Small samples swing. Moving from two mentions in five runs to three in five can easily be chance. Compare the same ten questions from the same places month to month, and judge the direction over several months, not one.

No one can promise you a placement. Google's own advice, covered in our guide to what Google says about GEO, llms.txt and AI SEO, treats visibility in Google's AI features as the same work as ordinary search optimization. Anyone who guarantees a mention in ChatGPT is guessing.

What our monthly AI tracking covers, and what it cannot

If you would rather not run 60 answers by hand, we do the asking as part of getting you found in AI answers. We put your questions to ChatGPT, Gemini, Grok and Google's AI answers from dozens of spots around your business, and record who gets recommended, how the assistant describes you and which sites it cites. You see the result in your monthly report, next to your Maps position.

It has limits. We sample answers, and we cannot see what any one visitor saw. We cannot reach private conversations or an assistant's memory of a user. We cannot make an assistant name you, and we do not promise it. We read the trend across months.

See where you stand today

Our free visibility map shows where you rank across your neighborhood today, the Maps half of the picture to set beside your AI log. It comes with no contract and no promise of a rank.

Why AI Recommendations Change Every Time, and How to Measure Yours