1

Should Feed/RSS be blocked via robots.txt?

Disallow: /*feed*
Disallow: /*rss*

If yes, is this the correct way to apply?

If you are sending Feed/RSS somewhere or if you have a subscription system, of course, you shouldn't do this. However, if you're like me and don't use any of these, should you block them?

When I look at GSC (Google Search Console), I see that URLs ending with /feed/ are also being added, and they lead to errors. Has this happened to you? I have a theory that this negatively affects my crawl budget and the overall SEO process.

Some sources say it's beneficial for robots to access additional information even if you don't use them, while others say you can turn them off if you don't use them.

A well-known and expert SEO professional has done it this way:

Disallow: */feed
User-agent: FeedFetcher-Google
Allow: /*feed
User-agent: YandexTurbo/1.0
Allow: /*feed
User-agent: Bingbot
Allow: /*feed

It's quite confusing. What do you think?

I know that there are two other questions similar to this, but I have checked them. That's not very explanatory for people who don't use Feed/RSS and who have GSC errors.

And they also don't explain how to disallow if chosen to.

Other questions:

1
  • Per Google, it seems like both have their own merits and uses in terms of crawling and indexing content. I would suggest, since you have access to the site's Search Console, submitting both as sitemaps. My one question is, if they aren't being used, why have them? developers.google.com/search/blog/2014/10/… Nov 8, 2023 at 16:26

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.