Pages

The Pages section lists everything the crawler has discovered. It is where you launch optimization, review the result and decide which content stays out of reach of AI agents.

Filters and search#

Two tabs split the content: All Pages and Excluded. Within the first, the Search by URL or title… field filters the list, and a row of chips narrows it by state:

Filter	What it shows
All	Every page that is not excluded.
Optimized	Pages already rewritten and ready for agents.
Need optimization	Scraped pages that have not been processed yet. This is the work queue.
In progress	Pages being processed right now.
Optimization failed	The AI rewrite failed.
Scraping failed	The crawler could not read the page.

Optimizing in bulk#

The Optimize All button opens a dialog with a custom optimization rules field. These are natural-language instructions applied to every page on the site during optimization, and they are saved for future runs, including auto-optimization runs.

Custom rules — example

Focus on B2B services and highlight the enterprise features.
Always mention the delivery time when the page states one.
Never invent prices: use only the ones in the original text.

Warning

Every processed page spends one optimization from the monthly package. On a 289-page site, Optimize All can drain the package in one go. Consider optimizing the pages that matter most first, by filtering and using the per-row Optimize button.

When failures have piled up, a Retry all button appears and reprocesses only the failed pages.

A page row#

Each row shows the page type, the title, the URL, the scrape and optimization dates, and a status label.

Status labels#

Label	Meaning
Pending	Discovered, not scraped yet.
Scraping…	The crawler is reading the page.
Scraped	Content stored, ready to optimize.
Optimizing…	The model is rewriting the page.
Optimized	Ready and visible to AI agents.
Scrape Failed	The content could not be read.
Optimize Failed	The rewrite failed.
Excluded	Out of the proxy and out of the discovery files.

Row actions#

Preview

Shows the optimized Markdown content inside the dashboard, with a View original link.

Preview AIO HTML

Shows the optimized HTML exactly as an AI crawler receives it. This is the definitive check that the proxy is serving the right thing.

Optimize

Processes this page alone. Spends one optimization.

Re-crawl source

Reads the page from your site again, without optimizing. Use it when you have edited the original content.

Update content

Re-optimizes the page with your feedback. A field opens where you can describe how you want it rewritten.

Exclude / Include

Takes the page out of the AIO system, or puts it back.

Tip

If you are not happy with the result for a page, there is no need to re-crawl: use Update content and describe the adjustment (“shorten the summary”, “mention the starting price”, “use a more technical tone”). Re-optimizing spends one optimization.

Excluded pages#

Excluding a page removes it from the AI optimization proxy and from the discovery files (llms.txt, robots.txt, sitemap.xml). It will not be optimized or indexed, and it will not be reachable through the AIO system.

It is the right tool for:

admin or internal-access pages;
checkout and cart flows;
legal pages duplicated across several languages;
any content you do not want to expose to AI agents.

The Excluded tab lists what is out and offers Include All to reverse it in bulk.

Referral traffic per page#

Opening a page also shows its LLM referral traffic: how many referrals it received over the last 30 days, by day or by month.

Note

The Unknown LLM platform groups visits whose AI source could not be identified: a URL pasted into the address bar, a bookmark, a link from a tool that strips the Referer header, or a private-mode browser. Clicks you make yourself from the dashboard are filtered out and never appear here.