Bot traffic

This section measures the mechanical half of the channel: which AI crawlers reach your site, how many requests they make and which paths they ask for. It is the direct proof that the integration works and that your content is being picked up.

Headline metrics#

Requests (7 d) and (30 d)

Total bot requests in each window.

Unique Bots

How many distinct crawlers have visited you.

Active Days

On how many days there was at least one visit.

Pages Visited

How many distinct paths were requested.

Crawlers by operator#

Groups visits by the company behind the crawler: OpenAI, Google, Anthropic, Perplexity, DeepSeek, Meta, Apple, ByteDance and Other, with the count and share of each.

Tip

This split tells you which AI ecosystems are picking your content up. An operator that is missing, or barely present, usually means its crawler is not getting through: check robots.txt and the configuration warnings.

Requests over time#

A trend chart of requests, with a period selector. The By Day and By Month filters change the aggregation, and All bots lets you isolate a single one. Show all and Hide all toggle the whole series selection.

Most popular paths#

A ranking of the paths bots request most, with their request counts. This is where you see which parts of your site the models actually care about.

Note

It is normal for /agentfi-sitemap to top the list: it is the sitemap AgentFi serves, and crawlers poll it regularly to spot changes. It is not an error.

Recent visits#

A chronological table of the latest recorded visits:

Column	Content
Bot	The identified user agent: GPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, OAI-SearchBot, Bytespider, Amazonbot, PetalBot…
Path	The requested URL.
Status	The HTTP status code of the response.
Time	Date and time of the visit.

Reading the status codes#

Code	How to read it
200	Fine. The bot got the page.
3xx	A redirect. If it repeats a lot, check the cookie-redirect configuration warning in Site overview.
4xx	The bot asked for something that does not exist or is blocked. A repeated pattern of 403s on valid paths points at firewall or `robots.txt` rules.
5xx	An error on your server during the bot's visit.

Index crawlers vs user-initiated agents#

Not every agent does the same thing, and the distinction changes the reading:

Index crawlers
GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot

They walk your site to build the model's index. Their volume tells you how much of your content is entering those systems.

User-initiated agents
ChatGPT-User, Claude-User, Perplexity-User

Someone is asking right now and the assistant went to fetch your page. They are the most direct signal of real demand.

That second group is also the basis of the estimated referral traffic calculation; see LLM referral traffic.

When there is no data#

If the section is empty, the message says there are no bot visits for this site yet and that statistics will appear once AI agents start visiting your optimized pages. Check, in this order:

that the integration is active;
that there are pages in the Optimized state;
that robots.txt does not block AI crawlers;
that enough time has passed — crawlers do not show up the same day.