Start with a useful website.
Make it easier to understand.

What should I work on first?

1. Make something useful. Solve a real problem and show how the tool works.
2. Check crawl access. Decide which search crawlers and training-related uses you want to allow.
3. Answer the question clearly. Give the reader a useful answer early, then add context and evidence.
4. Measure actual results. Use your webmaster tools and analytics to observe indexing, traffic, and conversions.

Search access and model training are separate controls.

A provider may use different crawlers for search discovery, model training, user-initiated retrieval, or link previews. Use the documented controls for each purpose.

*Google-Extended also controls certain Gemini grounding uses. Meta-ExternalAgent also has some product indexing uses. Review the provider’s documentation before selecting your preferences.

What does robots.txt actually do?

It communicates crawler preferences. It does not authenticate requests or force every crawler to cooperate. Some user-initiated fetchers may not follow these preferences. Use authentication for private content.

How do I install generated rules?

1. Back up your existing file. Compare each generated group with any group already present for that bot.
2. Merge the rules carefully. Preserve existing private-path restrictions and only include a sitemap that exists.
3. Publish the reviewed file at your website’s root: /robots.txt.
4. Check the live file and relevant crawl reports. Cached crawler preferences can take time to update.

What does the Cloudflare expression do?

The expression matches declared user-agent strings. It excludes requests to robots.txt so cooperating crawlers can read your policy. It cannot detect a crawler that spoofs a different agent. Google-Extended is a product token rather than a separate HTTP agent and is excluded from this expression. Review the expression in your Cloudflare account and choose the appropriate action there.

How the content grader works

The content grader runs in your browser. It checks five observable features and adds their weights for a score out of 100. This is our checklist, not a search engine’s scoring system.

The grader does not read the linked sources, verify your claims, check your live website, or measure search visibility. The definition check is a simple phrase heuristic. More words and links alone do not make a page useful.

Read the original sources

Provider behavior can change. These are the official pages used for the current crawler presets. Reviewed November 2, 2026.