I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like
User-agent: *
Disallow: /
User-Agent: GoogleBot
Allow: /
User-Agent: Bingbot
Allow: /
User-Agent: Slurp
Allow: /
User-agent: ia_archiver
Allow: /
Sitemap : [Sitemap URL]
I have also checked these search engines and they do show indexed results from my website. Also before I had added permission for Alexa's bot
User-agent: ia_archiver
Allow: /
without which Alexa was not able to do the audit. I am puzzled as to why Alexa is still reporting so, that all crawlers are blocked even when Alexa itself used that robots.txt permission to crawl my website.
I had already used Google's Robots.txt testing tool.
It reports that GoogleBot is not blocked.
Result when I feed a URL to a page in my website to be tested against selection of GoogleBot
I am using Allow operator as a way of having only selective bots be able to crawl my website. I would also like to highlight the fact that Alexa's own bot was not able to crawl my website for audit till I had added it to website's robots.txt using the same selective allow method.
User-agent
lines).