1

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule Disallow /wp-content in robots.txt). I waited for a couple of weeks and uploaded an actual file in the root with the same rules and it worked.

Does this mean that an actual file is better than dynamically generated robots.txt file or is it just me associating some coincidental events?

Improve this question
3
  • 4
    They both looks the same to the search engines. They can't tell if one is dynamic and another is not.
    – John Conde ♦
    Commented Apr 2, 2014 at 2:25
  • Are you sure the plugins folder was blocked in the original dynamic robots.txt? As the default dynamic robots.txt of wordpress does not block wp-content folder.
    – Max
    Commented Apr 2, 2014 at 7:24
  • @Max Yes, the company I work for has a custom wordpress package that generates a /robots.txt url (if robots.txt file is not present) with the rules recommended here: askapache.com/seo/updated-robotstxt-wordpress.html Didn't realise that wordpress does not have wp-content blocked by default, thanks for pointing that out, I have edited my question to avoid the confusion.
    – Anon
    Commented Apr 2, 2014 at 7:55

2 Answers 2

Reset to default
2

There is no difference to search engines between those two methods. You can generate it like you want.

Moreover, you need to know blocking a page with robots.txt is not the perfect way to prevent indexing of pages. You can read this question for more information.

Improve this answer
0

No difference at all as far as crawlers are able to access them (with all your valid rules) with HTTP 200 response.

Improve this answer
1
  • Please pay attention of posting an answer with additional information in relation to these already mentioned in other answers.
    – Zistoloen
    Commented Apr 3, 2014 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.