Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a private web server set up on a non-conventional port (say 6677). All people can have access to the whole site's urls from 6677. The links by mysite:6677 are also indexed by Google, which is not desireable, as I need to keep port 6677 discreet as a kind of workaround behind the front proxy which runs on 80.

So, how can I disallow access to 6677 in robots.txt?

share|improve this question
    
Why it is indexed? Does the mysite:6677 restricted by login page? (I guess yes since it's the site's admin). Then, why don't you just add propre meta tag to disallow indexing? –  j0k Sep 6 '12 at 12:11
    
No it is not restricted. people can see all the publically accessible urls from either of the ports. How can I add meta tags? –  alfish Sep 6 '12 at 12:26
    
Here is an example. –  j0k Sep 6 '12 at 12:34
    
@j0k assuming that the example meta work, it is undesirable since it still exposes the port 6677 in html code. –  alfish Sep 6 '12 at 14:20
    
Why does it expose the port? The HTML file has no knowledge of the port through which it is being accessed and there should be no need to mention it. Note that if you use the META tags then you need to allow robots.txt - as mentioned in my linked answer. –  w3d Sep 6 '12 at 14:25

1 Answer 1

up vote 4 down vote accepted

If you wish to disallow crawler access to example.com:6677 using robots.txt then you simply have to host your robots.txt file on the appropriate port, ie:

http://example.com:6677/robots.txt

The specification does not allow you to specify a port in the robots.txt file itself. Any paths specified use the same protocol, port number and host by which the file is accessed.

But, as mentioned in this answer, disallowing in robots.txt does not necessarily prevent the URL from being indexed; it prevents it from being crawled.


However, as noted in the comments, it seems that the same site is accessible from both port 80 and port 6677. But only port 6677 should be blocked from crawlers.

Since both ports access the same site then they would both share a common robots.txt file and so both sites would be blocked, unless you conditionally returned a different robots.txt file depending on which port was used to access the site. This could perhaps be done using .htaccess and an internal rewrite, but I don't think a robots.txt is what you require since you could still run the risk of these URLs being indexed.

In my opinion you need to conditionally check for the port in your server-side script and send the appropriate META tag or HTTP response header back to the client. In PHP you could do this with something like the following (near the top of your script):

<?php
// Block robots from port 6677
if ($_SERVER['SERVER_PORT'] == '6677') {
  header('X-Robots-Tag: noindex');
}
?>
share|improve this answer
    
robots.txt is already accessible through example.com:6677, just like any other publically accessible file of the site. –  alfish Sep 6 '12 at 13:48
    
Then your robots.txt file should already be preventing crawler access to these files, unless there is an error in the file? Is the link in SERPs a URL-only link, or is it complete with title and description? –  w3d Sep 6 '12 at 14:26
    
I'm not sure what SERP is. Just to clarify my aim: I don't wan ANY of the urls:6677/ become exposed in google search results. –  alfish Sep 6 '12 at 15:15
    
SERP - Search Engine Results Page. If you don't want the URLs to appear at all in the SERPs then don't disallow in robots.txt. You need to either add a noindex robots META tag to the page (as j0k suggests in comments) or return an X-Robots-Tag: noindex HTTP response header. If, however, you need to prevent any unauthorised access then you'll need to implement some kind of password authentication. –  w3d Sep 6 '12 at 15:28
    
Isn't there a way to do it only through robots.txt? –  alfish Sep 6 '12 at 22:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.