Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
I'm looking to understand a sitemap created a long time ago.
Is there a good reason to exclude Google's video crawler in robots.txt when you have a video site map?
Update: sorry just to be clear I do understand the differences between these, but I'm trying to understand a legacy robots.txt to see if my predecessors or an SEO agency may of had a reason to do this.
No, don't do that. Look's like you misunderstand about robots.txt and sitemap.
Video sitemap and other sitemap tells about your new content to Googlebot, so they come to your websites and start crawling, but before they crawl, they always check your robots.txt first. So If you disallow them, then they can't crawl anything.
Let me talk more about some of Google crawlers like Googlebot image and video. When Googlebot come to your site, and find out you have embed images then next time, Google image bot crawl that kind of data.
So, in general Googlebot crawl anything, but when they found specific content like images and videos, then next time they crawl with their specific crawler. It makes process fast for them.
Sitemap – list of pages of a web site accessible to crawlers or users
video sitemap - A video sitemap is a text file packed with useful data about the content of your video
robots.txt are signboards on your site giving signals to search engines that they are allowed to enter and crawl your site in order for them to collect and gather data.
However there are some robots.txt setup or configuration that will deny or won't allow search engine robots to crawl on specific pages of your site probably for privacy purposes, but this is configurable
Why would you exclude video?
You can use the Google Video extension to the Sitemap protocol to give Google information about video content on your site. You can create a new video Sitemap or add video information to an existing Sitemap.
