Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest.

I want to know if I add User-agent: Googlebot, will fake Googlebot crawlers be able to view my website? Is even possible to fake a bot?

share|improve this question

migrated from serverfault.com Dec 17 '13 at 23:35

This question came from our site for professional system and network administrators.

3 Answers 3

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything.

With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.

share|improve this answer
    
How can I know if the crawler is fake or not? –  AliBZ Dec 17 '13 at 23:03
1  
You don't .... The user agent is as reliable as a honest politician –  HBruijn Dec 17 '13 at 23:11
    
Is it possible to use the ip to check if its real or not? –  AliBZ Dec 17 '13 at 23:17
    
If the search engine in question publishes the netblocks they use for their crawlers, you can cross-reference that info with the IP you're wondering about. –  EEAA Dec 17 '13 at 23:18
    
It's impossible to know for sure since sometimes Google uses IP's that do not resolve nor registered with this information. –  bybe Dec 17 '13 at 23:55

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling:

User-Agent: *
Disallow: /

User-Agent: googlebot
Disallow:

User-Agent: bingbot
Disallow:

User-agent: slurp
Disallow:

Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or another legitimate crawler. Some crawlers impersonate browser user agents such as Internet Explorer or Firefox.

There is a procedure for verifying that a user agent of Googlebot is actually a Google crawler. It involves doing some DNS queries against the IP address from which the crawler visited.

There is also the concept of a spider trap which is a place on your website that users wouldn't find, but crawlers would. A spider trap can be used to identify crawlers that are masquerading as browser user agents.

share|improve this answer

As mentioned, user agents can be spoofed so it is unreliable for blocking access (in Google Chrome you can open Dev Tools and navigate to 'overrides' to change your user agent). No one with enough knowledge to spoof a major search engine's user agent is going to be deterred by robots.txt.

While it doesn't offer any protection from falsified user agents, adding directives to your server configuration files (e.g. .htaccess) to block user agents would offer you more protection...although I'm not sure why you'd want to do it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.