I am running a Mediawiki at the domain
someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively scanned by search engines with queries like:
or as well
where some_topic is an article of the wiki.
These requests seem to be of very low benefit but they consume a good deal of bandwidth and in addition I fear those automatically generated pages are not so good for my site's quality reputation in the serchengines evaluation.
As the requests a mostly done by 'good' engines such as Google or Bing I am quite sure they would obey robots.txt. So I added the following robots.txt to the folder of the base url
someurl.com (I added the whole robots.txt, even though only line 1 and 6 are relevant for the queries named above):
User-agent: * Disallow: User: Disallow: Discussion: Disallow: MediaWiki: Disallow: Special: Disallow: /login.php Disallow: /profile.php Disallow: /author/ Disallow: /category/ Disallow: /tag/
This robots.txt is active for about two days now and has been crawled already, but still there are many requests to URLs like the above which I thought blocked.
So I have the following questions now:
1) is the above logic correct and capable of denying access (to well behaving bots). Especially I wonder whether
Disallow: Spezial: correctly works as a wildcard to deny all request having "Special:" in URL or in parameter. I also wonder if the ":" in "Special:" might be a problem.
2) If so why then there is no effect yet? I consider I just have to grant more time to see the effect?
3) Will denying in the robots.txt lead to the de-indexing of this sites from the searchengines results? If not how can I get this huge amount of automatically generated URLs de-indexed?