2

Say I have a MediaWiki website and all the Special: namespace webpages are excluded with robots.txt Disallow: Special: but there are a few specific Special: webpages that I do want to include;

  • Allow:Special:RecentChanges
  • Allow:Special:RandomPage
  • Allow:Special:Categories

Is Allow: in robots txt an exclusion to an exclusion?

To ask a more specific and two-factored question: Is the code above what I need to add to robots.txt and is it correct to say that these allocations are "exclusions to the (general) exclusion"?

1 Answer 1

2

It is fair to say that the Allow directive is an exclusion from an exclusion, but that's all just semantics; you could also say that it's an "inclusion within an exclusion," or however you want to phrase it.

The point is, it allows you to make an exception for certain documents or folders when you've excluded their otherwise general path from being crawled by compliant spiders.

Your robots.txt will probably look more like this (depending on your URL structure; just make sure you start at the root):

# Your robots.txt
User-agent: *
Allow: /Special:RecentChanges
Allow: /Special:RandomPage
Allow: /Special:Categories
Disallow: /Special

I placed the Allow before Disallow - this format ensures that nothing's been disallowed before it could be allowed.

In short just follow the standards (Moz and Yoast and Google for reference), as they relate to your website. You have the right idea.

6
  • Hello, thank you for the answer ; please have a look in [mediawiki.org/wiki/Manual:Robots.txt] ; I understand that they do require : in that particular case and that this is not a directory ; please kindly consider to update the answer accordingly.
    – lahmedi
    Feb 23, 2021 at 8:35
  • I have suggested this edit (in case it doesn't appear): webmasters.stackexchange.com/review/suggested-edits/69782
    – lahmedi
    Feb 23, 2021 at 11:21
  • Hi @lahmedi, thank you for the additional info. I made some updates based on that. Please note, I don't know the full scope of your project, so the above robots.txt is just a basic example; you'll have to actually write the file it as you see fit, based on your structure of your actual URL's. Feb 23, 2021 at 15:06
  • 1
    "I put a wildcard after 'Categories,'" - robots.txt is prefix-matching anyway, so the * at the end of the URL-path is ignored.
    – MrWhite
    Feb 23, 2021 at 16:25
  • 1
    Thank you @MrWhite - amended, per your suggestion. Feb 23, 2021 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.