2

We had around 100 pages indexed which were actually outdated, thin content and causing SEO keyword cannibalisation issues.

For administrative purposes we are unable to remove the pages (as need them for internal purposes) but we have marked them all with meta name="robots" content="noindex" tags in the header

Given they are no longer linked anywhere on our website, and not part of our Sitemap, how will Google discover them to read the NoIndex directive and remove them from its index?

Improve this question

1 Answer 1

Reset to default
2

Google has already discovered them. Once Google knows about a URL Googlebot will periodically come back to fetch it. It will do so forever. I have URLs that have been "410 Gone" for a decade and Googlebot still occasionally fetches them.

The more relevant question is how long will it take until these URLs are next crawled. That depends on how popular the pages were. Since they were thin and duplicative my guess is that they weren't that popular. It may take Googlebot weeks or even months to come back and recrawl all of them.

You can speed up this process by creating a temporary XML sitemap that only contains the URLs that you want reprocessed. Submit it to Google through Google search console, and it should trigger Googlebot to visit those URLs more promptly, usually within a few days. Once you have verified that the pages have all be recrawled, you can remove the temporary sitemap.

Improve this answer
5
  • That’s a great response and you’re of course correct about the more relevant question! I had read however that sitemaps including no index URLs are risky even if temporary as a search engine’s definition of a sitemap is a file identifying URLs to be indexed … whereas this would be doing the reverse and would therefore potentially confuse the crawler. At least that’s what AHRefs were suggesting in a blog post. Any thoughts on that? I’m keen for these pages to be de-indexed asap! Checked today and 70% had been crawled and removed from index but 30% still remain … that’s 3 months in!
    – AK47onYou
    Oct 10 at 19:41
  • 1
    John Mueller from Google says you can use a temporary sitemap for this purpose: Need to expedite page removal in Google's search index? Try a temporary sitemap file
    – Stephen Ostermiller ♦
    Oct 10 at 20:35
  • Ah interesting ... never knew that but makes sense! I think given that out of the 100 or so pages we marked with meta no-index, now only around 30 or so remain as indexed, it's probably faster for us to use the GSC Temporary Removal also mentioned via the article you linked. I'm guessing this also forces Google to crawl the page again.
    – AK47onYou
    Oct 11 at 14:16
  • "forces" is a strong word. I'd say it strongly suggests that they should recrawl, and they usually do. But it isn't 100% guaranteed.
    – Stephen Ostermiller ♦
    Oct 11 at 15:00
  • Haha yes, not sure what made me write that. I'm pretty sure the number of people on this planet that can force Google to do anything is pretty small.
    – AK47onYou
    Oct 11 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.