For some context, I'm modifying somebody's website. This particular site is not yet published - it is accessible via direct IP.

Problem: Google has crawled the direct IP, and people are reaching the site. If we put a robots.txt into the root directory of the site, that would also prevent Google from indexing the site once it's published on an actual domain.

Is there any way using robots.txt to prevent search engines from indexing the site's pages over direct IP, but allow it to index the site via its domain name?

(If not, then what would be required is to serve a different robots.txt depending on what domain the site is running on. The server could look at the HOST header to do this, but what is the best way?)

  • Which web server are you using? I assume the domain name has a dedicated IP address and that you do not have a catch-all site. Make sure there are no IP address based links. Google does not like using IP addresses and prefers domain names. If your domain name has an dedicated IP address and no catch-all site, then the robots.txt would be the same for both domain and IP. It may be possible to use a modified version of @w3dk answer in this question: webmasters.stackexchange.com/questions/98464/… to present a different robos.txt file. – closetnoc Aug 18 '16 at 16:37

You need to do a 301 redirect from the IP address to the pages. Also use rel="canonical" microformat at the html head, here you state the url for every page.

  • 1
    rel=canonical is not a microformat but a HTML attribute. – Rob Sep 18 '16 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.