robots.txt, noindex and rel="canonical" are the three tools people reach for when they want a page to stay out of search. They are not interchangeable. Using the wrong one is one of the most common technical SEO mistakes, and it often makes the problem worse.
The one-line difference
| Tool | What it controls | Where it lives |
|---|---|---|
| robots.txt | Whether a crawler may fetch the URL | /robots.txt at the site root |
| noindex | Whether the page may appear in search results | A meta tag or X-Robots-Tag header |
| canonical | Which of several similar URLs is the preferred one | A <link> tag in the page head |
Crawling and indexing are two separate steps. A search engine first crawls a page (downloads it), then decides whether to index it (store it so it can be shown). Each tool acts on a different step.
robots.txt: stop the crawl
A rule like Disallow: /admin/ tells well-behaved crawlers not to request those URLs. It is good for keeping crawlers out of areas that waste their time: internal search results, cart and checkout steps, endless filter combinations.
It is not a way to remove a page from Google. A blocked URL can still be indexed if other sites link to it. It then appears in results with no description, because Google was never allowed to read the page.
noindex: allow the crawl, block the listing
noindex tells search engines to keep a page out of their results:
<meta name="robots" content="noindex">or, for files that are not HTML, as an HTTP header: X-Robots-Tag: noindex.
Use it for pages that must exist for visitors but should not appear in search: thank-you pages, internal tools, thin tag archives.
The mistake that makes noindex fail
If you block a page in robots.txt and add noindex, the crawler never fetches the page, so it never sees the noindex. The page can stay indexed. To remove a page, let it be crawled, add noindex, and only block it in robots.txt after it has dropped out of the index.
canonical: pick a winner among duplicates
The same content often lives at several URLs: with and without www, with tracking parameters, sorted or filtered versions. A canonical tag points to the one you want ranked:
<link rel="canonical" href="https://example.com/shoes/">Search engines treat it as a strong hint, not a command. It consolidates ranking signals onto one URL instead of splitting them. Every indexable page should carry a self-referencing canonical, meaning a canonical that points to itself.
Which one should I use?
- Keep crawlers away from a section (no need to rank it, saves crawl effort): robots.txt.
- Page should exist but not be in search: noindex, and leave it crawlable.
- Same content on several URLs: canonical to the preferred URL.
- Page is gone for good: return
404or410, or301redirect it to the closest replacement.
Test what a URL actually says
curl seoloop.in/indexable/example.com/some-page
curl seoloop.in/robots/example.comThe first answers yes or no with reasons (status code, robots meta, X-Robots-Tag, canonical). The second reads and analyses robots.txt. Run both before and after any change.
To check each one on a real page, see how to check a canonical tag, how to check robots.txt and how to check if a page is noindex.