I submitted my site to google, but I initially didn't have a robots.txt file. I added a robots.txt file disallowing some pages a few days later, but those pages are still in the index, e.g. when I do the query site:domain.com I still see thos pages. How can I make google remove those pages I just added to robots.txt?
3 Answers
You can remove a URL immediately and temporarily from the index by using Google's Removals Tool.
To keep it out of the index permanently, add a noindex html tag or http header.
Do NOT block the page with robots.txt, as this does not prevent indexing, it only prevents crawling. As per google:
If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. You can request that Google recrawl your page using the Fetch as Googletool. Another reason could also be that your robots.txt file is blocking this URL from Google web crawlers, so we can't see the tag.
answered Aug 1, 2020 at 6:28
The correct and only way is to initially allow to crawl the pages (remove the pages again from robots.txt). Set the meta tag name="robots" content="noindex,follow" on the affected pages.
Google will crawl the pages again and remove them from index after a certain time.
-
1I've downvoted this answer because it's never useful to have both a noindex tag and disallow in robots.txt at the same time. Google will ignore the noindex after a while, as it's disallowed from crawling it and seeing that it's still there. Commented Aug 1, 2020 at 6:16
-
I suggested to add the disallow in the robots.txt AFTER the bot has crawled the pages and de-indexed them. This is a common method!– MariusCommented Aug 15, 2020 at 10:40
-
I've never heard it once recommended anywhere to use both noindex and robots.txt blocking at the same time, and I've heard many places recommend against it. Commented Aug 15, 2020 at 11:31
-
You are still right with your statement. But i recommended the following: First, set noindex on the affected pages, THEN (when every page is de-indexed) use the robots.txt (disallow) to safe crawl-budget. To keep it clean it would be nice to remove then the noindex on the deindexed pages. But, google will igonre it, because of the disallow directive in robots.txt.– MariusCommented Aug 15, 2020 at 11:41
-
That's still not an accurate answer to the question, I'm afraid you're mistaken. It's not uncommon for a URL blocked by robots.txt to still be indexed by Google. Adding a page to robots.txt prevents Google from seeing the noindex tag anymore, so after a while it becomes fair game for indexing again. Commented Aug 15, 2020 at 11:49
You could also research whether there are any links out there pointing to the page you want to exclude? Then you could try uploading an updated sitemap on your Google Webmaster account. I would expect the whole process to take a while and a quick workaround would be implementing a 301 redirect for that particular page. Hope this helps.
-
1I downvoted this answer because sitemaps don't prevent pages from being indexed, and measuring incoming links is not an accurate way to determine or prevent indexation. I appreciate your answer and don't want to be unwelcoming, but it's just not very accurate! Commented Aug 1, 2020 at 6:20
-
@MaximillianLaumeister the question is about removing links from the google index not about blocking pages from being indexed. My solution is an alternative to getting google to update the index on a particular site. Commented Nov 17, 2020 at 10:49
-
"pages in Google's index" and "links from Google search results" are very similar concepts, and the way to make sure that a page stays removed is to block it from being indexed (not to be confused with blocking it from being crawled), so those goals go hand in hand. Commented Nov 18, 2020 at 2:50
robots.txtdoesn't keep pages from being indexed (this is a common misconception), only from being crawled. You will need to use arobots noindextag or header to prevent indexing.