robots.txt: View and Act on Crawl Directives and Blocked URLs

The robots.txt page shows your site’s robots.txt file as fetched by the app and lists crawled pages that have robots-related verification issues (e.g. blocked by robots, conflicting directives). Using it helps you avoid accidentally blocking important URLs and wasting crawl budget on the wrong pages.

Sections and how to use them

robots.txt content

The main section displays the raw robots.txt content for the property’s origin. You’ll see:

• User-agent rules — Which crawlers the rules apply to (e.g. * for all, Googlebot for Google).
• Allow / Disallow — Which paths are allowed or disallowed for crawling.
• Sitemap — Sitemap URLs declared in robots.txt (search engines use these for discovery).

How to use: Check that you’re not disallowing important sections (e.g. /blog/ or key landing pages). Ensure critical sitemaps are listed. If you use separate rules for different bots, verify they match your intent. Fix robots.txt on your server and re-check after the next crawl.

Pages with robots issues

A second section lists crawled pages that have robots-related verification issues. For example:

• Page is blocked by robots.txt but you expected it to be crawlable.
• Conflicting or ambiguous directives.
• X-Robots-Tag or meta robots issues that affect indexing.

Each row usually shows the page URL, title, and a short description of the issue. Pagination lets you move through the list.

How to use: Click a row to open the page in the page viewer for full details. Fix the robots.txt rule or the page-level directive (X-Robots-Tag, meta robots) so important pages are crawlable and indexable. For more on indexing directives, see X-Robots-Tag and indexing.

How the robots.txt page helps traffic and SEO

• Crawl budget — Correct Allow/Disallow rules steer crawlers to important URLs and away from low-value or duplicate areas, so more of your key pages get crawled and indexed.
• No accidental blocking — Catching “blocked by robots” issues on important URLs prevents lost traffic and rankings.
• Sitemap discovery — Listing sitemaps in robots.txt helps search engines discover all your sitemaps and index new or updated pages faster.
• Alignment with sitemaps — Use alongside the Sitemaps page so that URLs you want indexed are both in sitemaps and not disallowed in robots.txt.

Use robots.txt after site moves, template changes, or when you see unexpected “blocked” or “noindex” in Search Console. Fix directives, then re-crawl and re-check Verifications to confirm issues are resolved.