Say I have a MediaWiki website and all the Special: namespace webpages are excluded with robots.txt Disallow: Special: but there are a few specific Special: webpages that I do want to include;

  • Allow:Special:RecentChanges
  • Allow:Special:RandomPage
  • Allow:Special:Categories

Is Allow: in robots txt an exclusion to an exclusion?

To ask a more specific and two-factored question: Is the code above what I need to add to robots.txt and is it correct to say that these allocations are "exclusions to the (general) exclusion"?

New contributor
lahmedi is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

It is fair to say that the Allow directive is an exclusion from an exclusion, but that's all just semantics; you could also say that it's an "inclusion within an exclusion," or however you want to phrase it.

The point is, it allows you to make an exception for certain documents or folders when you've excluded their otherwise general path from being crawled by compliant spiders.

Your robots.txt will probably look more like this (depending on your URL structure; just make sure you start at the root):

# Your robots.txt
User-agent: *
Allow: /Special:RecentChanges
Allow: /Special:RandomPage
Allow: /Special:Categories
Disallow: /Special

I placed the Allow before Disallow - this format ensures that nothing's been disallowed before it could be allowed.

In short just follow the standards (Moz and Yoast and Google for reference), as they relate to your website. You have the right idea.

  • Hello, thank you for the answer ; please have a look in [mediawiki.org/wiki/Manual:Robots.txt] ; I understand that they do require : in that particular case and that this is not a directory ; please kindly consider to update the answer accordingly. – lahmedi Feb 23 at 8:35
  • I have suggested this edit (in case it doesn't appear): webmasters.stackexchange.com/review/suggested-edits/69782 – lahmedi Feb 23 at 11:21
  • Hi @lahmedi, thank you for the additional info. I made some updates based on that. Please note, I don't know the full scope of your project, so the above robots.txt is just a basic example; you'll have to actually write the file it as you see fit, based on your structure of your actual URL's. – Henry Visotski Feb 23 at 15:06
  • 1
    "I put a wildcard after 'Categories,'" - robots.txt is prefix-matching anyway, so the * at the end of the URL-path is ignored. – MrWhite Feb 23 at 16:25
  • 1
    Thank you @MrWhite - amended, per your suggestion. – Henry Visotski Feb 23 at 22:08

Your Answer

lahmedi is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.