New answers tagged robots.txt
2
The robots.txt spec does not allow for this - and even if it did, there are legal issues - a robot can not provide informed consent,/it does not have a mind so there can be no meeting of the minds. (If there is no meeting of the minds there is no contract)
Its likely to be an exercise in futility, but you could set up pages with arbitrary headers requiring ...
0
Old question, but current answers don't appear to have addressed the "error"(s) in the robots.txt file...
I have also put code in the robots.txt files to block access to the scripts directories, but these bots (Google, MS Bing, and Yahoo) ignore the rules and run the scripts anyways.
Yes, they are actually obeying the robots.txt file as written. ...
2
The ISP told me that this directive has only defined the crawl delay for the page /a-page-i-wanted-to-ignore
That does not make sense. The crawl-delay directive has nothing to do with the preceding Disallow directive.
The page /a-page-i-wanted-to-ignore shouldn't be crawled at all - since it's included in the Disallow directive, so if the crawl-delay should ...
3
Since you don't want bots to crawl these /search pages then you have no option other than to block them in robots.txt.
However, that doesn't necessarily prevent the /search pages from appearing as link only results in the SERPS (if other pages link to them) - but this is unlikely. And they are very unlikely to rank higher than other search results anyway.
2:...
1
Yoast's guide to robots.txt has the solution:
User-agent: *
Disallow: /*?*
(Replace the User-agent as necessary.)
The line above accounts for everything that comes after the first forward slash (after your root domain name), on either side of the question mark parameter delimiter, which needs to be present in the URL.
Top 50 recent answers are included
