I am running a Mediawiki at the domain someurl.com/wiki/
. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively scanned by search engines with queries like:
/index.php/Special:Whatlinkshere/some_topic
or as well
/index.php?title=Special:Whatlinkshere&target=some_topic
where some_topic is an article of the wiki.
These requests seem to be of very low benefit but they consume a good deal of bandwidth and in addition I fear those automatically generated pages are not so good for my site's quality reputation in the serchengines evaluation.
As the requests a mostly done by 'good' engines such as Google or Bing I am quite sure they would obey robots.txt. So I added the following robots.txt to the folder of the base url someurl.com
(I added the whole robots.txt, even though only line 1 and 6 are relevant for the queries named above):
User-agent: *
Disallow: User:
Disallow: Discussion:
Disallow: MediaWiki:
Disallow: Special:
Disallow: /login.php
Disallow: /profile.php
Disallow: /author/
Disallow: /category/
Disallow: /tag/
This robots.txt is active for about two days now and has been crawled already, but still there are many requests to URLs like the above which I thought blocked.
So I have the following questions now:
1) is the above logic correct and capable of denying access (to well behaving bots). Especially I wonder whether Disallow: Spezial:
correctly works as a wildcard to deny all request having "Special:" in URL or in parameter. I also wonder if the ":" in "Special:" might be a problem.
2) If so why then there is no effect yet? I consider I just have to grant more time to see the effect?
3) Will denying in the robots.txt lead to the de-indexing of this sites from the searchengines results? If not how can I get this huge amount of automatically generated URLs de-indexed?