I consider preventing indexation of all of MediaWiki 1.33.0's Special pages;
(In Hebrew, "מיוחד" means "special"):

Disallow: /מיוחד:*
Disallow: /index.php?title=מיוחד:*
Disallow: /index.php%3Ftitle%3D%D7%9E%D7%99%D7%95%D7%97%D7%93:*

Doing so is good in general because many of these pages aren't useful to the avarage surfer (rather to staff only) but some are important to both a regular users and crawlers.
A problem of not indexing "RecentChanges" and "Categories" is plausible though, as both these Special pages serve as "small, dynamic pseudo-sitemaps" that give access to virtually all webpages in the site.

Would you remove Disallow of MediaWiki special pages from robots.txt altogether?
Would you keep it with a good list exceptions just for "RecentChanges" and "Categories"?
Would you take a totally different approach?

  • RecentChanges and Categories are not useful as sitemaps. AllPages, maybe. – Tgr Aug 30 '19 at 16:43
  • Hi; please consider to elaborate more, thanks... – JohnDoea Aug 30 '19 at 16:50
  • They don't actually list all or even most pages of the wiki. RecentChanges might be useful for a search engine if it wants to learn about updates very quickly, but Google won't bother to do that with your wiki. – Tgr Aug 31 '19 at 8:34
  • I believe at might - but at least not at this point... – JohnDoea Aug 31 '19 at 9:12
  • I understand from your comment that Disallowing all special pages is bad for SEO anyway. But is it okay to keep all of them allowed? Isn't it bad as well? – JohnDoea Aug 31 '19 at 9:13

MediaWiki.org includes this in their robots.txt file:

Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Disallow: /wiki/Special%3A
Disallow: /wiki/Spezial%3A
Disallow: /wiki/Spesial%3A

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.