1

I added the no-index tag for the google-bot, but it is kind of taking a long time to de-index some pages since I have over 100k pages I need to de-index lol :)))

What is the proper way to de-index pages using robots.txt?

Thanks!

Any idea how long should it take?

Improve this question
1
  • I will recomended you remove page from Google webmaster tool.
    – Steven P
    Commented Aug 2, 2020 at 8:31

2 Answers 2

Reset to default
6

Assuming these pages still exist, but you just want them removed from search results...

What is the proper way to de-index pages using robots.txt?

You wouldn't necessarily use robots.txt to de-index pages. ie. remove already indexed pages from the Google search results. A noindex robots meta tag in the page itself (or X-Robots-Tag HTTP response header) might be preferable instead, in combination with the URL removal tool in Google Search Console (GSC) to speed up the process.

robots.txt specifically blocks crawling (not necessarily "indexing"). By blocking these pages from being crawled, these pages should naturally drop from the search index in time, but this can take a considerable amount of time. However, if these pages are still being linked to then they may not disappear entirely from the search results if these URLs are blocked by robots.txt (you can end up with a URL-only link in the SERPS, with no description).

Using robots.txt to remove the https://www.example.com/getridofthis/ directory...

User-agent: *
Disallow: /getridofthis/

To remove pages entirely from SERPs consider using a noindex meta tag (or X-Robots-tag: noindex HTTP response header) instead of robots.txt. (Which is what it sounds like you are doing already.) Don't block crawling in robots.txt as this will prevent the crawler from seeing the noindex meta tag.

To expedite the process of de-indexing URLs in Google search you can use the URL removal tool in GSC (formerly Webmaster Tools). For this tool to be effective long-term you need to use the noindex meta tag in the pages themselves. (The original blog article stated that robots.txt could be used as a blocking mechanism with the URL removal tool, however, recent help documents specifically warn against using robots.txt for "permanent removal".)

Reference:

Improve this answer
2
  • 1
    robots.txt prevents google from crawling and seeing noindex, so blocking both indexing and crawling with noindex and robots.txt is equivalent to only blocking crawling using robots.txt
    – Maximillian Laumeister ♦
    Commented Aug 1, 2020 at 6:47
  • 1
    @MaximillianLaumeister Very true. I've updated my answer to clarify that. Thanks. (In fact, I ended up reworking my answer. Some of the points in the original blog article I linked to with regards to robots.txt and the URL removal tool are no longer recommended in the more recent help documents.)
    – MrWhite
    Commented Aug 2, 2020 at 2:08
0

Take a look at robotstxt.org has all the info you need.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.