Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule Disallow /wp-content in robots.txt). I waited for a couple of weeks and uploaded an actual file in the root with the same rules and it worked.

Does this mean that an actual file is better than dynamically generated robots.txt file or is it just me associating some coincidental events?

share|improve this question
4  
They both looks the same to the search engines. They can't tell if one is dynamic and another is not. –  John Conde Apr 2 at 2:25
    
Are you sure the plugins folder was blocked in the original dynamic robots.txt? As the default dynamic robots.txt of wordpress does not block wp-content folder. –  Max Apr 2 at 7:24
    
@Max Yes, the company I work for has a custom wordpress package that generates a /robots.txt url (if robots.txt file is not present) with the rules recommended here: askapache.com/seo/updated-robotstxt-wordpress.html Didn't realise that wordpress does not have wp-content blocked by default, thanks for pointing that out, I have edited my question to avoid the confusion. –  vDog Apr 2 at 7:55

2 Answers 2

up vote 2 down vote accepted

There is no difference to search engines between those two methods. You can generate it like you want.

Moreover, you need to know blocking a page with robots.txt is not the perfect way to prevent indexing of pages. You can read this question for more information.

share|improve this answer

No difference at all as far as crawlers are able to access them (with all your valid rules) with HTTP 200 response.

share|improve this answer
    
Please pay attention of posting an answer with additional information in relation to these already mentioned in other answers. –  Zistoloen Apr 3 at 11:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.