3

I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like

User-agent: * 
Disallow: / 
User-Agent: GoogleBot 
Allow: / 
User-Agent: Bingbot 
Allow: / 
User-Agent: Slurp 
Allow: / 
User-agent: ia_archiver 
Allow: /
Sitemap : [Sitemap URL]

I have also checked these search engines and they do show indexed results from my website. Also before I had added permission for Alexa's bot

User-agent: ia_archiver
Allow: /

without which Alexa was not able to do the audit. I am puzzled as to why Alexa is still reporting so, that all crawlers are blocked even when Alexa itself used that robots.txt permission to crawl my website. Image of reported blocked crawlers

I had already used Google's Robots.txt testing tool. It reports that GoogleBot is not blocked. Link to result Result when I feed a URL to a page in my website to be tested against selection of GoogleBot

I am using Allow operator as a way of having only selective bots be able to crawl my website. I would also like to highlight the fact that Alexa's own bot was not able to crawl my website for audit till I had added it to website's robots.txt using the same selective allow method.

Improve this question
2
  • 1
    Though perfectly valid, blocking everything then allowing by agent is the opposite from how robots.txt was originally designed, used, documented, etc. Often, problems we see here with robots.txt code is because of this. It more difficult to get right. I am not suggesting that you change your code. I am suggesting that Alexa has a problem and you can test this by checking your code in as many places as you can. Alexa, the website, is pure junk. It may be that their SEO auditor is also junk. It would not surprise me. Cheers!!
    – closetnoc
    Jun 23, 2016 at 14:42
  • No idea if this has anything to do with it, but note that you must have line breaks between records (a record starts with one or more User-agent lines).
    – unor
    Jun 24, 2016 at 14:06

1 Answer 1

Reset to default
1

The default directive for spiders is allow and when Google visits your robots.txt and sees the disallow command, it immediately knows that it isn't allowed to crawl the website anymore. The subsequent allows don't matter.

Do note that when there are conflicting codes within the robots file or between the robots.txt and the Meta Robots tag of a page, spiders must follow the most restrictive rule.

If you want to block specific bots from crawling your website, you need to setup individual disallow rules for each of those bots.

On the other hand, if you want to allow some bots and disallow all the rest, then you can put the following:

User-agent: Googlebot
Disallow:

User-agent: Slurp
Disallow:

User-agent: *
Disallow: /

You can analyze your current robots.txt file with the built in checker in Google Search Console (former Webmaster Tools)

Improve this answer
4
  • Please, check my updated post above. I have posted result from the Google Search Console robots.txt tool. It reports no issues with the file.
    – HBalyan
    Jun 25, 2016 at 4:20
  • Is your website and its internal pages being indexed by Google?
    – pro_ng
    Jun 26, 2016 at 6:03
  • Yes.. they do get indexed and also come up in the SERP
    – HBalyan
    Jun 26, 2016 at 7:33
  • Then you may ignore the Alexa audit report
    – pro_ng
    Jun 26, 2016 at 7:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.