User-agent: * Disallow: /wp-admin Disallow: /sistema Disallow: /site Disallow: /old Sitemap: http://www.example.com.br/page-sitemap.xml
In the google test tool: https://www.google.com/webmasters/tools/robots-testing-tool is going normal: 'allowed'
But at ahref I have the feedback:
Fetching robots.txt took too long
I have already asked the hosting staff to see for us if there is any blocking of the crawler or the ips used by ahrefs and there is not.
I have already disabled all plugins on the site and nothing