2

On my request, our company has implemented a function in our website that adds <meta name="robots" content="noindex"> to blog articles and whitepaper pages that haven't been published yet.

Point is, as soon as the whitepaper or the blog article is published, the meta tag is removed. Does Google then automatically start crawling again? Or should we manually let Google know that there are new pages that should be crawled? If so, is there any way to let Google know automatically?

3

If you want to request indexing, you can submit it to Search Console. This tends to speed the process by manually indicating you'd like a page (and linked pages) considered for indexing.

If you remove tags, then you're waiting for Google to check the latest version. It will add your pages eventually if you remove tags.

The actual crawl speed varies greatly depending on factors that we aren't fully aware of but if I had to guess I would say Google likely makes 'noindex' pages lower priority. You can check your crawl frequency in Search Console as well to get an idea of how often Google updates itself on your content. If it looks high compared to other sites, it'd be worth experimenting with timing how long it takes for it to pick up one page that's had it's 'noindex' request removed.

  • Thanks for the quick response! I'm contemplating removing the tag from the unpublished articles and whitepapers, seeing as they're never on noindex for more than a few days. From my perspective, it's a worse downside to have to wait for Google to finally re-index the pages, than it is to have an article or whitepaper accidentally indexed. Thanks a lot! – Sean Apr 5 '17 at 8:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.