2

I submitted my site to google, but I initially didn't have a robots.txt file. I added a robots.txt file disallowing some pages a few days later, but those pages are still in the index, e.g. when I do the query site:domain.com I still see thos pages. How can I make google remove those pages I just added to robots.txt?

  • robots.txt doesn't keep pages from being indexed (this is a common misconception), only from being crawled. You will need to use a robots noindex tag or header to prevent indexing. – Maximillian Laumeister Aug 1 at 6:14
1

You can remove a URL immediately and temporarily from the index by using Google's Removals Tool.

To keep it out of the index permanently, add a noindex html tag or http header.

Do NOT block the page with robots.txt, as this does not prevent indexing, it only prevents crawling. As per google:

If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. You can request that Google recrawl your page using the Fetch as Googletool. Another reason could also be that your robots.txt file is blocking this URL from Google web crawlers, so we can't see the tag.

https://support.google.com/webmasters/answer/93710?hl=en

| improve this answer | |
0

The correct and only way is to initially allow to crawl the pages (remove the pages again from robots.txt). Set the meta tag name="robots" content="noindex,follow" on the affected pages. Google will crawl the pages again and remove them from index after a certain time.

| improve this answer | |
  • 1
    I've downvoted this answer because it's never useful to have both a noindex tag and disallow in robots.txt at the same time. Google will ignore the noindex after a while, as it's disallowed from crawling it and seeing that it's still there. – Maximillian Laumeister Aug 1 at 6:16
  • I suggested to add the disallow in the robots.txt AFTER the bot has crawled the pages and de-indexed them. This is a common method! – Marius Aug 15 at 10:40
  • I've never heard it once recommended anywhere to use both noindex and robots.txt blocking at the same time, and I've heard many places recommend against it. – Maximillian Laumeister Aug 15 at 11:31
  • You are still right with your statement. But i recommended the following: First, set noindex on the affected pages, THEN (when every page is de-indexed) use the robots.txt (disallow) to safe crawl-budget. To keep it clean it would be nice to remove then the noindex on the deindexed pages. But, google will igonre it, because of the disallow directive in robots.txt. – Marius Aug 15 at 11:41
  • That's still not an accurate answer to the question, I'm afraid you're mistaken. It's not uncommon for a URL blocked by robots.txt to still be indexed by Google. Adding a page to robots.txt prevents Google from seeing the noindex tag anymore, so after a while it becomes fair game for indexing again. – Maximillian Laumeister Aug 15 at 11:49
-2

You could also research whether there are any links out there pointing to the page you want to exclude? Then you could try uploading an updated sitemap on your Google Webmaster account. I would expect the whole process to take a while and a quick workaround would be implementing a 301 redirect for that particular page. Hope this helps.

| improve this answer | |
  • 1
    I downvoted this answer because sitemaps don't prevent pages from being indexed, and measuring incoming links is not an accurate way to determine or prevent indexation. I appreciate your answer and don't want to be unwelcoming, but it's just not very accurate! – Maximillian Laumeister Aug 1 at 6:20
  • @MaximillianLaumeister the question is about removing links from the google index not about blocking pages from being indexed. My solution is an alternative to getting google to update the index on a particular site. – Marius Cucuruz Nov 17 at 10:49
  • "pages in Google's index" and "links from Google search results" are very similar concepts, and the way to make sure that a page stays removed is to block it from being indexed (not to be confused with blocking it from being crawled), so those goals go hand in hand. – Maximillian Laumeister Nov 18 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.