1

I have a blog in Blogger, so the URL structure is always like this:

www.example.com/2019/01/post.html

The problem is I don't want google to crawl www.example.com/2019 (it shows all posts of 2019) and www.example.com/2019/01 (shows all posts of January).

Is there a way to block this in robots.txt, and allow the individual HTML posts to be crawled?

2

Frankly, you're better off not disallowing the two date-related categories in your robots.txt file, and using the noindex meta tags on these category pages (www.example.com/2019/ and www.example.com/2019/01/) instead. It will be less labor-intensive, and ensure that those pages are not in the SERPs. (A robots.txt directive does not guarantee that a page won't be indexed.) Just set the noindex robots meta tag on both category pages:

<meta name="robots" content="noindex, follow">

If you absolutely must use the robots.txt file for this, you can use the Allow directive, which is sometimes referred to as nonstandard in SEO literature. However, as MrWhite points out below, "exactly how the different bots interpret these non-standard additions to robots.txt could warrant further testing," plus you might still get those pages indexed, so the category page level meta tag would be preferable. If you still want to use the robots.txt file, you'd include something like this:

User-agent: *  
Allow: /2019/*.html$  
Disallow: /2019/

That would disallow anything in the 2019 folder, including folders for each month. But Googlebot would be able to access anything in that folder that ends with ".html". You'd have to update this every year to keep it current.

This was also addressed on Stack Overflow: https://stackoverflow.com/questions/42882200/robots-txt-disallow-folder-but-allow-files-within-folder

  • 1
    "not disallowing the two date-related categories" - I agree. In fact, I would think that if you disallowed the two date-related categories it would negatively affect the crawling/indexing of the "individual HTML posts", as those category pages link to the posts. – MrWhite Aug 6 at 22:13
  • 1
    "Allow is obeyed only by Googlebot" - Moz is incorrect/out-of-date here (and would seem to be using a non-authoritative reference?). Whilst Allow: is non-standard, it has been supported by most of the mainstream bots for many years. BingBot certainly does support the Allow: directive. Although exactly how the different bots interpret these non-standard additions to robots.txt could warrant further testing. – MrWhite Aug 6 at 22:15
  • 1
    @MrWhite Thank you for pointing out the error re: Allow directive. I should have done more research on that, especially since I relied on Moz for this detail, but Moz is unfortunately not what it used to be. I updated the response to reflect your info. – Henry Visotski Aug 6 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.