4

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site.

User-agent: Baiduspider
Disallow: /

User-agent: *
Disallow: /templates_c/
5

If you are using disallow:/ with user-agent: * then only entire site is blocked with robots.txt.

Otherwise only particular user-agent will get blocked with your sites.

Blocking user-agents:

The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).

To block the entire site, use a forward slash.

Disallow: /

To block a directory and everything in it, follow the directory name with a forward slash.

Disallow: /junk-directory/

To block a page, list the page.

Disallow: /private_file.html

To remove a specific image from Google Images, add the following:

User-agent: Googlebot-Image
Disallow: /images/dogs.jpg 

To remove all images on your site from Google Images:

User-agent: Googlebot-Image
Disallow: / 

To block files of a specific file type (for example, .gif), use the following:

User-agent: Googlebot
Disallow: /*.gif$

To prevent pages on your site from being crawled, while still displaying AdSense ads on those pages, disallow all bots other than Mediapartners-Google. This keeps the pages from appearing in search results, but allows the Mediapartners-Google robot to analyze the pages to determine the ads to show. The Mediapartners-Google robot doesn't share pages with the other Google user-agents. For example:

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /

For more details see this, and user-agents for Google here.

1
  • 2
    Thanks dan. That small piece of information means a lot to me thanks again. – madgrace Dec 5 '13 at 9:28
1

The Baiduspider bot is not allowed to crawl anything on your host.

All other bots are not allowed to crawl any URL whose path begins with /templates_c/. They may crawl all other URLs on your host.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.