I consider preventing indexation of all of MediaWiki 1.33.0's Special pages;
(In Hebrew, "מיוחד" means "special"):

Disallow: /מיוחד:*
Disallow: /index.php?title=מיוחד:*
Disallow: /index.php%3Ftitle%3D%D7%9E%D7%99%D7%95%D7%97%D7%93:*

Doing so is good in general because many of these pages aren't useful to the avarage surfer (rather to staff only) but some are important to both a regular users and crawlers.
A problem of not indexing "RecentChanges" and "Categories" is plausible though, as both these Special pages serve as "small, dynamic pseudo-sitemaps" that give access to virtually all webpages in the site.

Would you remove Disallow of MediaWiki special pages from robots.txt altogether?
Would you keep it with a good list exceptions just for "RecentChanges" and "Categories"?
Would you take a totally different approach?

  • RecentChanges and Categories are not useful as sitemaps. AllPages, maybe. – Tgr Aug 30 '19 at 16:43
  • Hi; please consider to elaborate more, thanks... – user58733 Aug 30 '19 at 16:50
  • They don't actually list all or even most pages of the wiki. RecentChanges might be useful for a search engine if it wants to learn about updates very quickly, but Google won't bother to do that with your wiki. – Tgr Aug 31 '19 at 8:34
  • I believe at might - but at least not at this point... – user58733 Aug 31 '19 at 9:12
  • I understand from your comment that Disallowing all special pages is bad for SEO anyway. But is it okay to keep all of them allowed? Isn't it bad as well? – user58733 Aug 31 '19 at 9:13

MediaWiki.org includes this in their robots.txt file:

Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Disallow: /wiki/Special%3A
Disallow: /wiki/Spezial%3A
Disallow: /wiki/Spesial%3A

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy