Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am making robots.txt dynamically instead of creating a physical robots.txt file. I am making this from the CMS and the system is appending a forward slash to every page. In the case of robots.txt, I have written the .htaccess code to redirect from mysite.com/robots.txt to mysite.com/robots.txt/.

Will the search engines have any problems getting to robots.txt in this way?

share|improve this question

migrated from stackoverflow.com Apr 25 at 15:10

This question came from our site for professional and enthusiast programmers.

    
What is need of adding slash after robots.txt? –  anubhava Apr 25 at 13:30
    
Its sort of a requirement to make this file from the CMS and the way CMS handles and displays pages, it add slashes no matter what and I am not allowed to change this functionality. –  HeroFTime Apr 25 at 13:41
    
Just to clarify, so mysite.com/robots.txt would not even be a valid request on your site/CMS? –  w3d Apr 25 at 16:27
    
See also: webmasters.stackexchange.com/questions/61089/… –  w3d Apr 25 at 18:36

3 Answers 3

For Google, the redirection should be OK.

3xx (redirection)
Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404. Handling of robots.txt redirects to disallowed URLs is undefined and discouraged. Handling of logical redirects for the robots.txt file based on HTML content that returns 2xx (frames, JavaScript, or meta refresh-type redirects) is undefined and discouraged.

Source: Robots.txt Specifications

However, if you have already successfully set up an external redirect, can you not change this to an internal rewrite? That way, the search engines won't even know the difference.

Although there is still the niggling thought... if robots.txt is not available directly I would try and resolve this with your CMS!?

share|improve this answer

I do not recommend a redirect but a rewrite. Google will follow a certain number of redirects (up to 5 per RFC specifications), but you cannot expect this of all bots.

The Google Developer's site provides details on Robots.txt Specifications. And specifically about how they process the results.

A Redirect would send the bot to a different URL. Using a RewriteRule you can serve

http://www.domain.com/robots.txt directly with a HTTP 200 response code.

In your case, you could use something like.

RewriteCond %{REQUEST_URI} ^/robots.txt$
RewriteRule .* /robots.txt/ [L]

If you do use this, you will need to assure that robots.txt does not actually exist, your Apache will use it instead. You may want to replace /robots.txt/ with something like index.php?robots or something unambiguous.

share|improve this answer

Most CMS systems use a .htaccess file with some rewrite rules to handle all the URLs on your site.

It is usually possible to add an exception to those rules such that certain files (such as robots.txt) are not handled by the CMS system.

An example of such a rule (from here) is:

RewriteCond %{REQUEST_URI} !^(/index\.php|/assets|/robots\.txt|/sitemap\.xml|/favicon\.ico)
RewriteRule ^(.*)$ /index.php/$1 [L]

The rewrite condition is what stops the following rewrite rule from taking effect. the ! means "not", the ^ is "starts with" and the | means "or". So the condition translates to: "the uri doesn't start with index.php, assets, robots.txt, sitemap.xml, or favicon.ico." Creating a similar rewrite condition in your .htaccess file should solve your problem.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.