2

My robots:

User-agent: *
Disallow: /wp-admin
Disallow: /sistema
Disallow: /site
Disallow: /old

Sitemap: http://www.example.com.br/page-sitemap.xml

In the google test tool: https://www.google.com/webmasters/tools/robots-testing-tool is going normal: 'allowed'

But at ahref I have the feedback:

Fetching robots.txt took too long

I have already asked the hosting staff to see for us if there is any blocking of the crawler or the ips used by ahrefs and there is not.

I have already disabled all plugins on the site and nothing

Improve this question
5
  • Is your robots.txt file a static file or dynamically generated at the time of request? – DocRoot Mar 21 '20 at 23:22
  • robots.txt is static. When accessing robots.txt through the browser, its content is always the same – sNniffer Mar 22 '20 at 7:12
  • Maybe the problem is with ahrefs servers/location? To clarify, GSC/Google isn't actually reporting any problem with your robots.txt file? – DocRoot Mar 25 '20 at 12:02
  • Just wondered... is the robots.txt request being redirected from HTTP to HTTPS? – DocRoot Mar 25 '20 at 17:40
  • Both, but the site was at http – sNniffer Apr 2 '20 at 21:15
0

I changed hosting and right after I tested the ahrefs crawler and everything worked perfectly, probably something from the host was blocking ahrefs ips.

Incredible is that the hosting staff did not know how to solve it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.