Managing sites
The AI Optimization screen lists every site you have access to. This is where you add new sites, re-run crawls and open each one in detail.
Adding a site#
- Click Add Site
The button sits at the top right, next to the Sites quota counter.
- Enter the domain URL
Including the protocol:
https://example.com. Use the root domain, not an inner page. - Set a page cap, if you want one
Optional field. Limits how many pages the crawler will discover. Leave it empty to use your plan's limit.
- Decide whether the crawl starts now
Start crawling immediately is on by default.
- Decide about weekly tests
Enable weekly auto-tests runs your saved visibility queries every week. Recommended, though every run spends tokens.
If you have reached your plan's site count, the button opens a limit notice instead of the form. The per-tier limits are in Plans, tokens and optimizations.
The site card#
Each site shows up as a card with its favicon, the domain, a status indicator and a line of facts: page count, sitemap presence and the date of the last crawl.
Possible states#
| Status | Meaning |
|---|---|
| Pending | The site was created but has not been crawled yet. |
| Crawling | A crawl is running. The card reports progress. |
| Ready | The crawl finished; the pages are available. |
| Active | The site is live with the integration working. |
| Not Verified | The domain has not been verified yet. Crawling and optimization work; the proxy deployment does not. |
| Error | The last crawl failed. See Troubleshooting. |
The card menu#
The three-dot icon opens two actions:
- Re-crawl — runs page discovery again. Use it after publishing new content or changing the structure of the site. Crawling spends neither optimizations nor tokens.
- Delete — asks for a second confirmation (Confirm delete) before removing the site.
Deleting a site also deletes its pages, its visibility queries and its entire result history. The action cannot be undone. If you only want to stop processing it, turn off auto-optimization and the scheduled tests instead of deleting it.
Filtering the list#
The Filter by domain… field narrows the list as you type. If nothing matches, you get No sites match your filter.
When the list is empty#
There are two different messages, and they are worth telling apart:
Re-crawling: when it is needed#
| Situation | Crawl again? |
|---|---|
| You published new pages | Yes. The crawl discovers them and leaves them Scraped. |
| You edited the text of an existing page | Re-crawl source on that row in Pages is enough. |
| You changed the sitemap | Yes. |
| You want to optimize pages that are already scraped | No. Go straight to Optimize All. |
| Auto-optimization is on | Usually not: it revisits the sitemap at whatever cadence you set. |