Discovery files

Discovery files are how AI crawlers get into your site. This section decides, file by file, whether AgentFi generates and serves them or leaves yours untouched.

llms.txt Management#

llms.txt is a Markdown index of your pages and offerings, designed specifically for language models. There are two modes:

Mode	What it does	When to pick it
Managed by AgentFi	AgentFi generates and serves `llms.txt` with your offerings and your optimized pages.	The default. It is what you want in almost every case.
Use site's original	AgentFi will not proxy `llms.txt`. Your site serves its own file.	You already maintain an `llms.txt` by hand and would rather not delegate it.

Warning

If the mode is Managed but there is no optimized page yet, AgentFi cannot generate the file. The section is flagged with a warning triangle in the site navigation bar, and the notice tells you to rebuild the site or optimize pages first.

Preview and rebuild#

Two tabs — llms.txt and llms-full.txt — show the generated content, with a Copy button and an Open button that loads it from your site. The progress indicator shows how many pages are included.

Rebuild regenerates the file from scratch.

Danger

Rebuilding re-optimizes every page through the LLM. It can take several minutes and spend a significant number of optimizations and tokens. The dashboard asks for an explicit confirmation before it starts.

The custom instructions field steers the generation:

Instructions for llms.txt

Focus on B2B services, emphasize enterprise features.

robots.txt#

AgentFi detects whether your site already has a robots.txt and offers the same two modes: Managed by AgentFi or Use site's original.

The Re-detect button checks the file on your site again. Use it if you have just published or changed it.

Note

If the panel says the content was not saved, click Re-detect to refresh it.

sitemap.xml#

AgentFi looks for your sitemap and reports what it found:

Found and indexed

The sitemap was located and its URLs are in the system. The detected URL is shown on the card itself.

Not found on site

There is no sitemap.xml. Page discovery will be far more limited.

SSL certificate warning

If the sitemap was detected using relaxed TLS, the panel warns about an SSL certificate problem: AI bots with strict certificate validation may not be able to reach your site at all. It is worth reviewing your domain's certificate chain.

Open Knowledge Format (OKF)#

OKF is an open standard from Google Cloud: a bundle of interlinked Markdown knowledge concepts, one per page, that AI agents can walk. AgentFi generates it from your optimized pages and links it from llms.txt so crawlers can discover it.

State	What happens
Enabled	AgentFi generates and publishes the OKF bundle at `/okf/` and advertises it everywhere.
Disabled	AgentFi does not serve an OKF bundle for this site.

The bundle index preview shows the content of /okf/index.md, and the Open button loads it from your site.

Tip

Leave OKF enabled unless you have a specific reason not to. It is an extra discovery channel that does not compete with llms.txt or the sitemap: it complements them.

Original Discovery Files#

The Original Discovery Files card summarizes what AgentFi found on your site before stepping in, with the exact path of every detected file. It is the reference for deciding which mode to apply to each one.

A practical check#

Once the modes are configured, confirm everything is served correctly by opening these URLs in a browser:

URLs to check

https://your-domain.com/llms.txt
https://your-domain.com/llms-full.txt
https://your-domain.com/robots.txt
https://your-domain.com/sitemap.xml
https://your-domain.com/okf/index.md

If any of them fails to answer, check Integration: the proxy has to be active for managed files to be served.