SEO

Noindex vs Robots.txt: Keeping Pages Out of Google

By Kathy P. Carr · 2026-10-11 · 3 min read

Which should you use: noindex or robots.txt?

If your goal is to keep a page out of Google Search results, use noindex and leave the page crawlable. Robots.txt is designed to manage crawling, not to control what appears in the index. Mixing the two is one of the most common ways a page ends up in search results when its owner wanted it gone. This article summarizes what Google's documentation and the Robots Exclusion Protocol standard say, so you can pick the right tool for each situation.

What does robots.txt actually do?

According to Google's introduction to robots.txt, the file mainly manages crawler traffic, largely to avoid overloading a site with requests. Google states plainly that it is not a mechanism for keeping a web page out of Google.

The same documentation explains the catch: Google will not crawl content blocked by robots.txt, but it may still index the blocked URL if other sites link to it. In that case the URL, and possibly anchor text from those links, can appear in results, and the listing will not have a description because Google could not read the page.

What does noindex do, and how do you set it?

Google's guidance on blocking indexing describes noindex as a rule you set with either a robots meta tag in the page's HTML or an HTTP response header. Once Googlebot sees the rule, it drops the page from Search results, even if other sites link to it.

• Meta tag: a robots meta tag with the value noindex in the page head works for HTML pages.
• HTTP header: an X-Robots-Tag header with noindex works for HTML and for non-HTML files such as PDFs or images.

You can combine noindex with other directives, and Google also documents a Googlebot-specific version of the meta tag if you want to target only that crawler.

Why does blocking and noindex together fail?

Googlebot has to fetch a page to read its noindex rule. Google's documentation says that if a robots.txt file blocks the page, the crawler will never see the noindex rule, and the page may still show up in results if it is linked elsewhere.

A typical failure looks like this: a site owner adds noindex to a set of thin pages, then also disallows that folder in robots.txt, believing two layers are safer. The disallow prevents Google from ever re-reading the pages, so the noindex is never processed. The practical fix, based on Google's guidance, is to remove the disallow, let the pages be crawled, wait until they drop out of results, and only then consider a block if crawl load is a real concern.

Google also notes that it does not honor a noindex rule placed inside the robots.txt file itself. Any older tutorial suggesting that approach is out of date for Google.

Is robots.txt a security measure?

No. The Robots Exclusion Protocol, standardized as RFC 9309, states that its rules are not a form of access authorization. They are requests that well-behaved crawlers are expected to honor, not barriers that enforce anything. The RFC also points out that the file is publicly readable, so listing a sensitive path in it advertises that path to anyone who looks. For content that must stay private, the RFC points toward application-layer controls such as HTTP authentication.

Google's own list of alternatives for keeping a URL out of search results agrees: password-protect the page or files on your server, use a noindex meta tag or header, or remove the page entirely.

A simple decision guide

One caveat: these sources describe Google's behavior and the general standard. Other search engines and AI crawlers may handle directives differently, and compliance with robots.txt is voluntary for any crawler. Check each platform's own documentation if a specific engine matters to you.

Frequently asked questions

Does robots.txt remove a page from Google?

No. Google says robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still be indexed if other sites link to it, typically without a description.

Can I use noindex and disallow on the same page?

It defeats the purpose. If robots.txt blocks the page, Googlebot cannot fetch it and never sees the noindex rule, so the page may stay in results.

Does Google support noindex inside robots.txt?

No. Google's documentation says a noindex rule placed in robots.txt is not supported. Use a robots meta tag or an X-Robots-Tag HTTP header instead.

How do I keep a private page truly private?

Use authentication such as a password. RFC 9309 says robots.txt rules are not access authorization, and the file is publicly readable.

Sources

1. Block Search indexing with noindex — Google Search Central
2. Introduction to robots.txt — Google Search Central
3. RFC 9309: Robots Exclusion Protocol — IETF / RFC Editor

Keep reading