Search Results
| Search type | Search syntax |
|---|---|
| Tags | [tag] |
| Exact | "words here" |
| Author |
user:1234 user:me (yours) |
| Score |
score:3 (3+) score:0 (none) |
| Answers |
answers:3 (3+) answers:0 (none) isaccepted:yes hasaccepted:no inquestion:1234 |
| Views | views:250 |
| Code | code:"if (foo != bar)" |
| Sections |
title:apples body:"apples oranges" |
| URL | url:"*.example.com" |
| Saves | in:saves |
| Status |
closed:yes duplicate:no migrated:no wiki:no |
| Types |
is:question is:answer |
| Exclude |
-[tag] -apples |
| For more details on advanced search visit our help page | |
Results tagged with robots.txt
Search options not deleted
user 95153
Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
2
votes
1
answer
2k
views
Ahrefs reports "Fetching robots.txt took too long"
/www.example.com.br/page-sitemap.xml
In the google test tool: https://www.google.com/webmasters/tools/robots-testing-tool is going normal: 'allowed'
But at ahref I have the feedback:
Fetching robots.txt …
- 121
0
votes
Accepted
Ahrefs reports "Fetching robots.txt took too long"
I changed hosting and right after I tested the ahrefs crawler and everything worked perfectly, probably something from the host was blocking ahrefs ips.
Incredible is that the hosting staff did not k …
- 121