Recently I got email message from Google that Googlebot
cant access your site
. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file being set as User-agent: * Disallow: /
so does was mine.
How can I allow Goolebot
and Bingbot
to crawl the content of my site? I am going to put as below but I am not sure. Am I in the right track?
User-agent: googlebot
User-agent: google
User-agent: bingbot
User-agent: bing
User-agent: *
Disallow: /