My robots:
User-agent: *
Disallow: /wp-admin
Disallow: /sistema
Disallow: /site
Disallow: /old
Sitemap: http://www.example.com.br/page-sitemap.xml
In the google test tool: https://www.google.com/webmasters/tools/robots-testing-tool is going normal: 'allowed'
But at ahref I have the feedback:
Fetching robots.txt took too long
I have already asked the hosting staff to see for us if there is any blocking of the crawler or the ips used by ahrefs and there is not.
I have already disabled all plugins on the site and nothing
robots.txt
file a static file or dynamically generated at the time of request? – DocRoot Mar 21 '20 at 23:22robots.txt
is static. When accessing robots.txt through the browser, its content is always the same – sNniffer Mar 22 '20 at 7:12robots.txt
file? – DocRoot Mar 25 '20 at 12:02robots.txt
request being redirected from HTTP to HTTPS? – DocRoot Mar 25 '20 at 17:40