This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site.
User-agent: Baiduspider Disallow: / User-agent: * Disallow: /templates_c/
Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.Sign up to join this community
If you are using
user-agent: * then only entire site is blocked with robots.txt.
Otherwise only particular user-agent will get blocked with your sites.
Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (
To block the entire site, use a forward slash. Disallow: / To block a directory and everything in it, follow the directory name with a forward slash. Disallow: /junk-directory/ To block a page, list the page. Disallow: /private_file.html To remove a specific image from Google Images, add the following: User-agent: Googlebot-Image Disallow: /images/dogs.jpg To remove all images on your site from Google Images: User-agent: Googlebot-Image Disallow: / To block files of a specific file type (for example, .gif), use the following: User-agent: Googlebot Disallow: /*.gif$ To prevent pages on your site from being crawled, while still displaying AdSense ads on those pages, disallow all bots other than Mediapartners-Google. This keeps the pages from appearing in search results, but allows the Mediapartners-Google robot to analyze the pages to determine the ads to show. The Mediapartners-Google robot doesn't share pages with the other Google user-agents. For example: User-agent: * Disallow: / User-agent: Mediapartners-Google Allow: /
Baiduspider bot is not allowed to crawl anything on your host.
All other bots are not allowed to crawl any URL whose path begins with
/templates_c/. They may crawl all other URLs on your host.