5

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages;
(In Hebrew, "מיוחד" means "special"):

Disallow: /מיוחד:*
Disallow: /index.php?title=מיוחד:*
Disallow: /index.php%3Ftitle%3D%D7%9E%D7%99%D7%95%D7%97%D7%93:*

Doing so is good in general because many of these pages aren't useful to the avarage surfer (rather to staff only) but some are important to both a regular users and crawlers.
A problem of not indexing "RecentChanges" and "Categories" is plausible though, as both these Special pages serve as "small, dynamic pseudo-sitemaps" that give access to virtually all webpages in the site.

Would you remove Disallow of MediaWiki special pages from robots.txt altogether?
Would you keep it with a good list exceptions just for "RecentChanges" and "Categories"?
Would you take a totally different approach?

6
  • RecentChanges and Categories are not useful as sitemaps. AllPages, maybe.
    – Tgr
    Commented Aug 30, 2019 at 16:43
  • Hi; please consider to elaborate more, thanks...
    – user58733
    Commented Aug 30, 2019 at 16:50
  • They don't actually list all or even most pages of the wiki. RecentChanges might be useful for a search engine if it wants to learn about updates very quickly, but Google won't bother to do that with your wiki.
    – Tgr
    Commented Aug 31, 2019 at 8:34
  • I believe at might - but at least not at this point...
    – user58733
    Commented Aug 31, 2019 at 9:12
  • I understand from your comment that Disallowing all special pages is bad for SEO anyway. But is it okay to keep all of them allowed? Isn't it bad as well?
    – user58733
    Commented Aug 31, 2019 at 9:13

1 Answer 1

2

MediaWiki.org includes this in their robots.txt file:

Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Disallow: /wiki/Special%3A
Disallow: /wiki/Spezial%3A
Disallow: /wiki/Spesial%3A
1
  • So presumably if that do it it isn't horrible practice. Commented Sep 23, 2019 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.