10

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files for a single domain, as far as I know.

Could I put a line in .htaccess

Redirect 301 /robots.txt /robots.php

(or equivalent instruction, and please, indicate which one if allowed)

so it redirects to a dynamic php file, where I can serve different contain according to the $_SERVER['HTTP_HOST']?

And the same question for sitemap.xml, so I can serve a dynamic sitemap.php which indicates different links for each different domain.

The problem with no using .txt and .xml is, as mentioned, that all the domains share a single physical directory in the server computer.

11

You can make any file dynamic. The best way to do so is not through redirects, but through rewrite rules.

RewriteRule ^robots\.txt$  /robots.php [L]

That way, you power it with a dynamic script, but the URL doesn't change. Most crawlers (including Googlebot) will follow redirects for robots.txt, but some crawlers will get confused if you introduce redirects.

Note that even if you power it with PHP, your robots.txt should appear to be static to each crawler for each domain. It is fine to serve different content for different domains, or even for different user agents. However, serving different content randomly, or based on time of day can really confuse search engine crawlers and mess up your SEO.


Sitemaps are fine to name however you want. You could redirect those, or use a rewrite rule to power them dynamically at the same URL. You can also name them like

  • site-a-sitemap.xml
  • site-b-sitemap.xml
  • site-c-sitemap.xml

Then refer to them in robots.txt:

Sitemap: http://www.example.com/example-sitemap.xml

or submit them to the search engines manually through their webmaster tools or search console.

  • Thanks to both for your response. Please correct what might be a typo, it is w3d instruction which worked, so the code should be RewriteRule ^robots\.txt$ robots.php [L] without the \ symbol. – Cesar Oct 22 '15 at 10:39
  • Yes, the version with the slash would be appropriate for your apache.conf file. For .htaccess, you need to leave it off. I have edited the answer to include the appropriate version for .htaccess. – Stephen Ostermiller Oct 22 '15 at 13:07
  • @Cesar The slash prefix on the pattern (ie. ^/robots\.txt$) would be required if this directive was in the server config, but yes, it will not match in per-directory .htaccess files. The slash prefix on the substitution (ie. /robots.php) is optional in this case. – MrWhite Oct 22 '15 at 13:07
5

Yes, the same way any request can be "dynamic".

However, you would not redirect (as in your example code), you should internally rewrite using mod_rewrite. (The same as what Drupal is probably already doing.)

For example, in your root .htaccess file:

RewriteEngine On
RewriteRule ^robots\.txt$ robots.php [L]

RewriteEngine should only occur once (although it doesn't really matter if it occurs multiple times).

You just have to make sure that it doesn't conflict with any other directives in your .htaccess file. So, this should probably be near the start of the file, certainly before your front controller.

4

Making the sitemap file dynamic is fine -- it's a good way to auto-update your sitemaps.

Making the robots.txt file dynamic (for the same host! Doing this for separate hosts is essentially just a normal robots.txt file for each of them.) would likely cause problems: it's not crawled every time a URL is crawled from the site, so it can happen that the "wrong" version is cached. For example, if you make your robots.txt file block crawling during business hours, it's possible that it's cached then, and followed for a day -- meaning nothing gets crawled (or alternately, cached when crawling is allowed). Google crawls the robots.txt file about once a day for most sites, for example.

  • I see no difference here among static or dynamic. I would also use the dynamic part to offer different versions according to different hosts, but because the hosts all share the same physical directory in the computer server, that is a way of having robots1.txt, robots2.txt, robots3.txt (numbers meaning in which domain we are). – Cesar Oct 28 '15 at 7:26
  • I don't think that dynamic here means that they want to serve different content each time. They just want to power it via PHP so they can make decisions based on the host name in PHP code. I often make robots.txt dynamic to serve different rules to different user agents. – Stephen Ostermiller Oct 29 '15 at 18:21
  • 2
    Yeah, as I mentioned, doing it for multiple hosts is essentially like having separate robots.txt files per host, which is fine. However, we sometimes see sites trying to control crawling by time of day using a dynamic robots.txt file -- that causes a lot of problems. – John Mueller Nov 2 '15 at 9:48
  • Good point. I've edited my accepted answer with a warning not to make robots.txt very dynamic. – Stephen Ostermiller Nov 6 '15 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.