I am making robots.txt dynamically instead of creating a physical robots.txt file. I am making this from the CMS and the system is appending a forward slash to every page. In the case of robots.txt, I have written the .htaccess code to redirect from mysite.com/robots.txt to mysite.com/robots.txt/.

Will the search engines have any problems getting to robots.txt in this way?

  • What is need of adding slash after robots.txt?
    – anubhava
    Apr 25 '14 at 13:30
  • Its sort of a requirement to make this file from the CMS and the way CMS handles and displays pages, it add slashes no matter what and I am not allowed to change this functionality.
    – HeroFTime
    Apr 25 '14 at 13:41
  • Just to clarify, so mysite.com/robots.txt would not even be a valid request on your site/CMS?
    – MrWhite
    Apr 25 '14 at 16:27
  • See also: webmasters.stackexchange.com/questions/61089/…
    – MrWhite
    Apr 25 '14 at 18:36

For Google, the redirection should be OK.

3xx (redirection)
Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404. Handling of robots.txt redirects to disallowed URLs is undefined and discouraged. Handling of logical redirects for the robots.txt file based on HTML content that returns 2xx (frames, JavaScript, or meta refresh-type redirects) is undefined and discouraged.

Source: Robots.txt Specifications

However, if you have already successfully set up an external redirect, can you not change this to an internal rewrite? That way, the search engines won't even know the difference.

Although there is still the niggling thought... if robots.txt is not available directly I would try and resolve this with your CMS!?


I do not recommend a redirect but a rewrite. Google will follow a certain number of redirects (up to 5 per RFC specifications), but you cannot expect this of all bots.

The Google Developer's site provides details on Robots.txt Specifications. And specifically about how they process the results.

A Redirect would send the bot to a different URL. Using a RewriteRule you can serve

http://www.domain.com/robots.txt directly with a HTTP 200 response code.

In your case, you could use something like.

RewriteCond %{REQUEST_URI} ^/robots.txt$
RewriteRule .* /robots.txt/ [L]

If you do use this, you will need to assure that robots.txt does not actually exist, your Apache will use it instead. You may want to replace /robots.txt/ with something like index.php?robots or something unambiguous.


Most CMS systems use a .htaccess file with some rewrite rules to handle all the URLs on your site.

It is usually possible to add an exception to those rules such that certain files (such as robots.txt) are not handled by the CMS system.

An example of such a rule (from here) is:

RewriteCond %{REQUEST_URI} !^(/index\.php|/assets|/robots\.txt|/sitemap\.xml|/favicon\.ico)
RewriteRule ^(.*)$ /index.php/$1 [L]

The rewrite condition is what stops the following rewrite rule from taking effect. the ! means "not", the ^ is "starts with" and the | means "or". So the condition translates to: "the uri doesn't start with index.php, assets, robots.txt, sitemap.xml, or favicon.ico." Creating a similar rewrite condition in your .htaccess file should solve your problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy