Guide · AI search
Is ChatGPT sending you visitors?
AI assistants reach your site in two ways: their crawlers read your pages, and their answers send people who click. Your analytics sees the second, if you know where to look, and almost never the first. This guide shows how to measure both, what to allow, and what actually helps.

Visitors: check referrers
Look forchatgpt.com,perplexity.ai,claude.ai,gemini.google.comandcopilot.microsoft.comas sources.Crawlers: check your server
AI crawlers don’t run JavaScript, so no analytics script sees them. Their visits are only in server logs.Allow answers, decide on training
Search crawlers (likeOAI-SearchBot) get you cited. Training crawlers (likeGPTBot) are a separate choice.
Two kinds of AI traffic
Someone asks ChatGPT for “a simple uptime tool for side projects”. Before that answer existed, a crawler fetched pages across the web. While answering, ChatGPT may fetch a few more live. Then it shows a list with links, and some people click.
That’s two different things reaching your site, and they show up in different places:
- Crawlers are programs fetching your pages: for training a model, for a search index, or on behalf of one person’s question right now. They read and leave. No analytics script sees them.
- Referred visitors are people who clicked a link in an AI answer. They’re real visits, with a referrer like
chatgpt.com, and your analytics counts them like any other.
Crawlers are the cause; visitors are the effect. To know whether AI search is working for you, you want both: who reads you, and who comes.
- Crawler readsA search crawler like
OAI-SearchBotfetches your pages. - Assistant answersYour page is in its index, or fetched live for the question.
- Link is citedThe answer lists your site with a link.
- Person visitsThey click, and arrive with a referrer like
chatgpt.com.
1. Count the people AI sends you
Start here: it works with any analytics tool, today. Open your sources (or referrers) for the last 30 days and look for these:
| Assistant | Source to look for | Notes |
|---|---|---|
| ChatGPT | chatgpt.com, sometimes chat.openai.com | Links in ChatGPT search results are often tagged utm_source=chatgpt.com too. That’s observed behaviour, not documented by OpenAI. |
| Perplexity | perplexity.ai | Cites sources on every answer, so it’s often the biggest of these for small sites. |
| Claude | claude.ai | Mostly links people click from answers that used web search. |
| Gemini | gemini.google.com | Google’s AI Overviews in Search arrive as google.com, not as Gemini. |
| Copilot | copilot.microsoft.com | Bing results arrive as bing.com. |
Group them, then look past the count
One or two visits a day from each won’t stand out in a list sorted by visitors. Group them into one channel (“AI assistants”) and watch the line over months, not days.
Then ask the better question: what do these people do? Someone who clicked a cited link already asked a specific question, so they tend to land deep (a docs page, pricing) and convert better than the average visitor. Compare their signup rate and, if you link payments to visits, their revenue, with your other channels.
Why the real number is higher than you see
Some AI traffic arrives with no referrer at all: mobile apps, desktop apps and copy-pasted links send none. Those visits land in “Direct”. If Direct grew on a page that only an AI answer would link to (a deep docs page, a comparison), that’s a hint.
You can’t fix that for links others share, but you can for links you control: tag the links you put in your own llms.txt, README or docs with utm_source.
2. See which AI crawlers read you
Each AI company runs several crawlers with different jobs. The names matter, because robots.txt rules and your decisions are per crawler. As each company documents them (checked Nov 15, 2026):
| Crawler | Company | What it’s for |
|---|---|---|
OAI-SearchBot | OpenAI | Shows websites in ChatGPT’s search results. Blocked sites aren’t shown in ChatGPT search answers. |
GPTBot | OpenAI | Training OpenAI’s models. Independent of search. |
ChatGPT-User | OpenAI | Fetches a page when a person’s request needs it. Not automatic crawling; OpenAI says robots.txt rules may not apply. |
Claude-SearchBot | Anthropic | Improves Claude’s search results. |
ClaudeBot | Anthropic | Collects content that may be used for training. |
Claude-User | Anthropic | Fetches pages when someone asks Claude a question. |
PerplexityBot | Perplexity | Links websites in Perplexity’s search results. Perplexity says it isn’t used for training. |
Perplexity-User | Perplexity | Fetches for a person’s request; Perplexity says it generally ignores robots.txt for those. |
Google-Extended | A robots.txt token, not a crawler: whether Googlebot’s crawl may train Gemini. No effect on Google Search. | |
Applebot-Extended | Apple | A robots.txt token for training Apple’s models. Pages that block it can still appear in search. |
0
What your analytics script sees of them
Vercel measured AI crawler traffic across its network in 2024 and found that none of the major AI crawlers render JavaScript. They fetch the HTML and leave, so a browser-based analytics tool can’t count them. Their visits are only in your server’s logs.
Where to find them
- Your host’s logs. Vercel, Netlify and Cloudflare show requests with their user agent. Filter for the names above.
- Your own server. A few lines in the code every request passes through can count them. This one only logs; send the line wherever you keep logs.
Next.js: count AI crawlers in proxy.ts (middleware.ts before Next.js 16)
proxy.ts import { NextResponse, type NextRequest } from "next/server"; const AI = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Meta-ExternalAgent|Bytespider|CCBot|Amazonbot/i; export function proxy(req: NextRequest) { const ua = req.headers.get("user-agent") ?? ""; const bot = ua.match(AI)?.[0]; if (bot) console.log(JSON.stringify({ ai_crawler: bot, path: req.nextUrl.pathname })); return NextResponse.next(); } // Pages only: skip assets and API routes. export const config = { matcher: ["/((?!_next/|api/|.*\\..*).*)", "/robots.txt", "/llms.txt"] };Express (or anything with a response hook): log the status too
server.js app.use((req, res, next) => { const bot = (req.get("user-agent") ?? "").match(AI)?.[0]; if (bot) res.on("finish", () => console.log(JSON.stringify({ ai_crawler: bot, path: req.path, status: res.statusCode }))); next(); });The status code tells you what they got. A crawler collecting
404s or403s is reading a site that isn’t there for it.
Check it’s really them
A user agent is just a string. OpenAI (openai.com/searchbot.json, gptbot.json, chatgpt-user.json), Anthropic (claude.com/crawling/bots.json) and Perplexity (perplexity.ai/perplexitybot.json, perplexity-user.json) publish the addresses their crawlers use. If a decision depends on it, check the address.
3. Choose who may read: robots.txt
Robots.txt is where you tell crawlers what they may fetch. With AI crawlers there are really two decisions:
- Answers and search (
OAI-SearchBot,Claude-SearchBot,PerplexityBot, plus Googlebot and Bingbot). Block these and you won’t be cited, so you won’t get the visitors from step 1. For most indie products, allow them. - Training (
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot). Blocking these keeps your pages out of future model training and doesn’t affect search. That’s a values call, not an SEO one.
Here’s a robots.txt that allows answers and opts out of training:
# Search and answers: welcome
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Model training: no thanks
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else
User-agent: *
Disallow: /app/
Sitemap: https://yourapp.com/sitemap.xmlCheck what you’re already blocking
Many sites block AI crawlers without knowing it:
- A template or plugin added
Disallow: /for a list of bots. - Your CDN or firewall blocks them. Cloudflare has an AI Crawl Control dashboard with per-crawler Allow and Block. A blocked crawler gets a
403, which robots.txt won’t show you. - Bot protection (challenges, rate limits) treats crawlers like attackers.
Open yourapp.com/robots.txt and read it as a crawler would: the group naming the crawler wins, otherwise * applies. Then try a fetch with a crawler’s name and look at the status:
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.0" https://yourapp.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot/1.0" https://yourapp.com/pricing4. Make pages easy to read
Being allowed in isn’t the same as being understood. Crawlers that don’t run JavaScript only see what your server sends.
Put the words in the HTML
If your pricing, features or docs are rendered in the browser (a client-only React page), a crawler sees an empty shell. Server-render or statically generate the pages you want cited. Quick test: curl https://yourapp.com/pricing and search the output for your prices.
Answer one question per page
AI answers quote pages that state something plainly. A page titled “Uptime monitoring for side projects” that says what it does, for whom and at what price in its first paragraph is easier to cite than a clever hero line. Comparison pages and guides with real numbers work well, because people ask comparison questions.
Keep the basics right
- A
sitemap.xmllisted in robots.txt. - Pages that answer
200fast, with no login or cookie wall in front. - A clear
<title>and description on every page. They’re often what gets quoted next to the link.
llms.txt: cheap, but don’t expect miracles
llms.txt is a proposed standard: a short Markdown file at /llms.txt that tells agents what your site is and where the important pages are. The only required part is an H1 with your site’s name; a summary and lists of links follow.
No AI company says its crawlers read it yet. Coding agents do use it when you point them at a site, and it takes ten minutes, so it’s worth having. Just don’t treat it as a ranking switch.
# YourApp
> YourApp watches uptime, traffic and revenue for every app you ship, in one list. For solo founders with several side projects.
## Docs
- [Get started](https://yourapp.com/docs): add an app and the script in two minutes
- [Pricing](https://yourapp.com/pricing): plans from $12 a month, 14-day trial
## Compare
- [YourApp vs Plausible](https://yourapp.com/compare/plausible-alternative)5. All of it in one place
Everything above works with logs, a spreadsheet and patience. BugOtter puts it on one screen per app, under Analytics → Go deeper → Bots:
- AI assistants: for ChatGPT, Claude, Perplexity and the others, how often their crawlers read your site next to how many visitors they sent, how many of those reached a goal, and the page they landed on. The visitor side works with the script alone.
- Crawlers that read your site: about 50 crawlers by name, grouped into AI, search, SEO tools and link previews, with the pages each one read. This needs a few lines on your server; the Bots view has a prompt your coding agent can follow, and it never slows a response down.
- What crawlers find: BugOtter reads your robots.txt and llms.txt once a day and tells you if robots.txt blocks a crawler that feeds AI answers. With status codes from your server, it lists the pages crawlers get
404on and the crawlers being turned away. - A milestone: the first visitor ChatGPT, Claude or Perplexity sends an app gets its own card to share.
Add the script
One tag in your
<head>, no cookies by default. AI visitors show up from the first click, grouped as “AI assistants”.Open Bots and copy the prompt
Your agent adds the crawler report to your server (Next.js, Express, Django, Rails, Laravel…). Test it with
curl -A "GPTBot/1.2" https://yourapp.com/: it shows up within a minute.Read it weekly, not daily
AI traffic is small and lumpy. Look at the trend and at what those visitors do, and fix what crawlers trip over first.
Pitfalls
- Reading crawls as visibility. A crawler reading a page doesn’t mean it’s cited. Training crawlers read everything and send nobody. The number that matters is visitors and what they do.
- Blocking the wrong bot. Blocking
GPTBotto “stay out of ChatGPT” does nothing for ChatGPT search; blockingOAI-SearchBotdoes. Know which one you mean. - Trusting the user agent. Scrapers dress up as GPTBot. Check the published address lists before acting on a spike.
- Client-only pages. If the words aren’t in the HTML, there’s nothing to cite, however good the page looks in a browser.
- Expecting it to be big. For most small sites AI assistants are a small channel today. The point of measuring it is to see it grow, and to catch the day a robots.txt change or firewall rule cuts it off.
Crawler names and purposes checked against OpenAI’s, Anthropic’s, Perplexity’s, Google’s and Apple’s own crawler pages on Nov 15, 2026. JavaScript finding: Vercel, “The rise of the AI crawler” (Dec 2024). llms.txt: llmstxt.org. Companies change these often; check their pages before acting on a detail.
Questions
Does blocking GPTBot keep me out of ChatGPT?
No. GPTBot is OpenAI’s training crawler. ChatGPT’s search uses a different one, OAI-SearchBot, and OpenAI says each setting is independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Block OAI-SearchBot and your site won’t be shown in ChatGPT search answers.
Does blocking Google-Extended hurt my Google ranking?
No. Google says Google-Extended doesn’t affect inclusion in Google Search and isn’t a ranking signal. It controls whether your content is used to train Gemini models and for grounding in Gemini Apps and Vertex AI. It’s a robots.txt token only: no crawler visits with that name.
Why don’t I see GPTBot in my analytics?
Analytics scripts run in the browser, and the major AI crawlers don’t run JavaScript (Vercel measured this in 2024 for OpenAI’s, Anthropic’s and Perplexity’s crawlers). They fetch the HTML and leave, so a script-based tool never hears about them. You need server logs, your host’s logs, or a small report from your server.
Do AI crawlers read llms.txt?
No AI company says so officially. llms.txt is a proposal (llmstxt.org) for a short Markdown file that tells agents what your site is. OpenAI, Anthropic and Google publish one for their own docs, but none of their crawler pages mention reading it. It’s cheap to add and coding agents use it when you point them at it; just don’t expect it to change your traffic on its own.
Can I trust a user agent that says GPTBot?
Not on its own: anyone can send that string. OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use (JSON files linked below). If it matters, check the request’s address against those lists before you treat it as the real crawler.
How many visitors should I expect from AI assistants?
For most small sites, few: a handful a week at first, often more from Perplexity and ChatGPT search than from chat. What makes them interesting is what they do next. People who click a cited link already asked a specific question, so check their signups and sales, not just the count.
See who AI sends you, and who it reads.
14 days of everything, no card needed. Plans from $10 a month, billed yearly.