I submitted my site to google, but I initially didn't have a robots.txt file. I added a robots.txt file disallowing some pages a few days later, but those pages are still in the index, e.g. when I do the query site:domain.com I still see thos pages. How can I make google remove those pages I just added to robots.txt?

  • robots.txt doesn't keep pages from being indexed (this is a common misconception), only from being crawled. You will need to use a robots noindex tag or header to prevent indexing. – Maximillian Laumeister Aug 1 at 6:14

You can remove a URL immediately and temporarily from the index by using Google's Removals Tool.

To keep it out of the index permanently, add a noindex html tag or http header.

Do NOT block the page with robots.txt, as this does not prevent indexing, it only prevents crawling. As per google:

If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. You can request that Google recrawl your page using the Fetch as Googletool. Another reason could also be that your robots.txt file is blocking this URL from Google web crawlers, so we can't see the tag.


| improve this answer | |

The correct and only way is to initially allow to crawl the pages. Set the meta tag name="robots" content="noindex,follow" on the affected pages. When the pages were removed from index, THEN add the disallow in the robots.txt.

In your actual setting you are telling google only this: "Please do not access/recrawl these pages". How should Google know that you want to deindex these pages?

| improve this answer | |
  • 1
    I've downvoted this answer because it's never useful to have both a noindex tag and disallow in robots.txt at the same time. Google will ignore the noindex after a while, as it's disallowed from crawling it and seeing that it's still there. – Maximillian Laumeister Aug 1 at 6:16

You could also research whether there are any links out there pointing to the page you want to exclude? Then you could try uploading an updated sitemap on your Google Webmaster account. I would expect the whole process to take a while and a quick workaround would be implementing a 301 redirect for that particular page. Hope this helps.

| improve this answer | |
  • 1
    I downvoted this answer because sitemaps don't prevent pages from being indexed, and measuring incoming links is not an accurate way to determine or prevent indexation. I appreciate your answer and don't want to be unwelcoming, but it's just not very accurate! – Maximillian Laumeister Aug 1 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.