2

My robots:

User-agent: *
Disallow: /wp-admin
Disallow: /sistema
Disallow: /site
Disallow: /old

Sitemap: http://www.example.com.br/page-sitemap.xml

In the google test tool: https://www.google.com/webmasters/tools/robots-testing-tool is going normal: 'allowed'

But at ahref I have the feedback:

Fetching robots.txt took too long

I have already asked the hosting staff to see for us if there is any blocking of the crawler or the ips used by ahrefs and there is not.

I have already disabled all plugins on the site and nothing

Improve this question
5
  • Is your robots.txt file a static file or dynamically generated at the time of request?
    – DocRoot
    Commented Mar 21, 2020 at 23:22
  • robots.txt is static. When accessing robots.txt through the browser, its content is always the same
    – sNniffer
    Commented Mar 22, 2020 at 7:12
  • Maybe the problem is with ahrefs servers/location? To clarify, GSC/Google isn't actually reporting any problem with your robots.txt file?
    – DocRoot
    Commented Mar 25, 2020 at 12:02
  • Just wondered... is the robots.txt request being redirected from HTTP to HTTPS?
    – DocRoot
    Commented Mar 25, 2020 at 17:40
  • Both, but the site was at http
    – sNniffer
    Commented Apr 2, 2020 at 21:15

1 Answer 1

Reset to default
0

I changed hosting and right after I tested the ahrefs crawler and everything worked perfectly, probably something from the host was blocking ahrefs ips.

Incredible is that the hosting staff did not know how to solve it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.