1

I'm looking to understand a sitemap created a long time ago.

Is there a good reason to exclude Google's video crawler in robots.txt when you have a video site map?

Update: sorry just to be clear I do understand the differences between these, but I'm trying to understand a legacy robots.txt to see if my predecessors or an SEO agency may of had a reason to do this.

2

No, don't do that. Look's like you misunderstand about robots.txt and sitemap.

Video sitemap and other sitemap tells about your new content to Googlebot, so they come to your websites and start crawling, but before they crawl, they always check your robots.txt first. So If you disallow them, then they can't crawl anything.

Let me talk more about some of Google crawlers like Googlebot image and video. When Googlebot come to your site, and find out you have embed images then next time, Google image bot crawl that kind of data.

So, in general Googlebot crawl anything, but when they found specific content like images and videos, then next time they crawl with their specific crawler. It makes process fast for them.

  • Thanks, I'll mark yours as the answer, I do understand the differences just that I'm trying to understand why someone would chose to do this in the past. – Alex KeySmith Aug 18 '16 at 8:19
2
  1. Sitemap – list of pages of a web site accessible to crawlers or users

  2. video sitemap - A video sitemap is a text file packed with useful data about the content of your video

  3. robots.txt are signboards on your site giving signals to search engines that they are allowed to enter and crawl your site in order for them to collect and gather data.

However there are some robots.txt setup or configuration that will deny or won't allow search engine robots to crawl on specific pages of your site probably for privacy purposes, but this is configurable

1

Why would you exclude video?

You can use the Google Video extension to the Sitemap protocol to give Google information about video content on your site. You can create a new video Sitemap or add video information to an existing Sitemap.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.