0

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file being set as User-agent: * Disallow: / so does was mine.

How can I allow Goolebot and Bingbot to crawl the content of my site? I am going to put as below but I am not sure. Am I in the right track?

User-agent: googlebot
User-agent: google
User-agent: bingbot
User-agent: bing

User-agent: *
Disallow: /
  • You are looking at this backwards. Normally, you allow all and block specific bots. It is not possible to use the robots.txt to block rogue scraper bots. Only valid bots will read the robots.txt file. This means that you can only block those who follow the rules and behave well. If you simply empty out your robots.txt file and block unwanted bots as you find them, you will be fine. This is how it is normally done. – closetnoc Sep 17 '15 at 23:41
3

No, the last two lines of your robots.txt file take precedence over the first four as the syntax of those first four lines is incorrect. As a result Google is blocked from viewing your website.

To allow Google and Bing you must specifically and individually allow each crawler:

User-agent: googlebot
Disallow: 

User-agent: bingbot
Disallow: 

User-agent: *
Disallow: /

Going forward, to find out if your robots.txt file will allow Google to index your site test it inside of your Google Webmaster Tools account. Also see the official robots.txt documentation.

  • John, What makes a differnce what if i leave it as User-agent: * Disallow: /? – Aaron Sep 18 '15 at 2:09
  • I'm not sure what you're asking here. – John Conde Sep 18 '15 at 2:10
  • Ok what is the differece between User-agent: googlebot Disallow: User-agent: bingbot Disallow: User-agent: * Disallow: / the one on your solution and just User-agent: * Disallow: / is there any difference between the two? the third one allows all bots including google and bing – Aaron Sep 18 '15 at 2:12
  • The first two lines allow googlebot to crawl your site. Same for bingbot on the next two. The last lines excludes all others who aren't those two bots. – John Conde Sep 18 '15 at 2:13
  • ok so the last line blocks all robots while the first two line allow google and bing to crawl my site respectively is that right? so googlebot and bingbot will see my site – Aaron Sep 18 '15 at 2:17
1

Based on the contents of your robots.txt file and your needs, you're better off making that file entirely empty or even better, maybe with a link to the sitemap files so search engines can find your URLs.

I suggest this because crawlers normally think they are allowed to access anything. By specifically using Disallow, you're trying to tell at least one crawler that they are not allowed to access something on your site.

Here's a link to an example of a robots.txt that google uses:

http://www.google.ca/robots.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.