How to check robots.txt and test a URL

robots.txt is a small text file that tells crawlers which parts of a site they may fetch. One wrong line can hide an entire site from search, and it is easy to do by accident when a staging setting reaches production.

Find the file

It always lives at the root of the host:

https://example.com/robots.txt

Open that in a browser. Note that rules apply per host and protocol. https://example.com/robots.txt does not cover https://blog.example.com/ or http://example.com/. Each needs its own file.

From a terminal, SEO Loop fetches and analyses it:

curl seoloop.in/robots/example.com

Real output for seoloop.in:

User-agent: *
Allow: /

Sitemap: https://seoloop.in/sitemap.xml

How to read the rules

A file is made of groups. Each group starts with a User-agent line, then Allow and Disallow rules:

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

Sitemap: https://example.com/sitemap.xml

The points that trip people up:

• Disallow: with nothing after it blocks nothing. Disallow: / blocks the entire site.
• Rules match from the start of the path. Disallow: /search blocks /search, /search?q=x and /search-results.
• When rules conflict, the most specific (longest) matching rule wins. That is why Allow: /search/help works above.
• * matches any characters and $ marks the end of a URL. Disallow: /*.pdf$ blocks PDF files.
• Google ignores Crawl-delay. Some other crawlers honour it.
• The Sitemap: line is how crawlers discover your sitemap. It can appear anywhere in the file.

Test whether a specific URL is blocked

Pick the URL, then find the group that applies to the crawler, then the longest matching rule. Doing that by hand is error-prone, so use a tool:

• Google Search Console has a robots.txt report under Settings. It shows the file Google fetched, when, and any errors. Its URL Inspection tool also tells you if a URL is blocked.
• From the terminal, curl seoloop.in/indexable/example.com/some-page answers yes or no and lists the reasons, including a robots.txt block.

What the server response does

The way the file is served matters as much as its content:

• 200: the rules are read and obeyed.
• 404 or another 4xx: treated as "no restrictions". Crawlers assume everything is allowed.
• 5xx (server error): Google pauses crawling the site, because it cannot tell what is allowed. A robots.txt that errors out can quietly stop your site being crawled.

Keep the file small. Google reads only the first 500 KiB.

Mistakes that cost traffic

• Disallow: / left over from staging. The classic. Check it after every launch.
• Blocking CSS and JavaScript. Google needs these to render your pages. Blocking them can make pages look broken to the crawler.
• Using robots.txt to hide pages from search. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. Use noindex for that, and let the page be crawled. See robots.txt vs noindex vs canonical.
• No sitemap line. Not an error, but an easy win. See how to check a sitemap.

Checklist

• File loads at the root with a 200
• No unintended Disallow: /
• CSS, JS and images are not blocked
• Sitemap URL listed
• Re-checked after every deploy