3

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest.

I want to know if I add User-agent: Googlebot, will fake Googlebot crawlers be able to view my website? Is even possible to fake a bot?

Improve this question
0

3 Answers 3

Reset to default
6

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything.

With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.

Improve this answer
5
  • How can I know if the crawler is fake or not?
    – AliBZ
    Dec 17, 2013 at 23:03
  • 1
    You don't .... The user agent is as reliable as a honest politician
    – HBruijn
    Dec 17, 2013 at 23:11
  • Is it possible to use the ip to check if its real or not?
    – AliBZ
    Dec 17, 2013 at 23:17
  • If the search engine in question publishes the netblocks they use for their crawlers, you can cross-reference that info with the IP you're wondering about.
    – EEAA
    Dec 17, 2013 at 23:18
  • It's impossible to know for sure since sometimes Google uses IP's that do not resolve nor registered with this information.
    – Simon Hayter
    Dec 17, 2013 at 23:55
6

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling:

User-Agent: *
Disallow: /

User-Agent: googlebot
Disallow:

User-Agent: bingbot
Disallow:

User-agent: slurp
Disallow:

Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or another legitimate crawler. Some crawlers impersonate browser user agents such as Internet Explorer or Firefox.

There is a procedure for verifying that a user agent of Googlebot is actually a Google crawler. It involves doing some DNS queries against the IP address from which the crawler visited.

There is also the concept of a spider trap which is a place on your website that users wouldn't find, but crawlers would. A spider trap can be used to identify crawlers that are masquerading as browser user agents.

Improve this answer
1
  • For SEO, you probably want to imitate the big boys and put your Disallow: / last, like li, fb, etc
    – hobs
    Apr 26, 2016 at 21:25
1

As mentioned, user agents can be spoofed so it is unreliable for blocking access (in Google Chrome you can open Dev Tools and navigate to 'overrides' to change your user agent). No one with enough knowledge to spoof a major search engine's user agent is going to be deterred by robots.txt.

While it doesn't offer any protection from falsified user agents, adding directives to your server configuration files (e.g. .htaccess) to block user agents would offer you more protection...although I'm not sure why you'd want to do it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.