11

I have a subomain which I do not want to be listed in any search engine result. Let say I have:

  • http://www.example.com
  • http://child.example.com

How can I hide all URLs of child.example.com domain which are currently showing in the search engine results?

|improve this question

migrated from superuser.com Jan 10 '13 at 7:46

This question came from our site for computer enthusiasts and power users.

14

In the root of the directory of the subdomain website, add a file called robots.txt containing:

User-agent: *
Disallow: /

This will tell web crawlers not to index the site at all. They do not have to obey, but the main ones will.

|improve this answer
  • Hi, i believe this is the best thing. And one more thing let me know, if i do that, how long will it take to effect on the Web Results? (e.g. Google Search) Do i need time or will it take suddenly? – 夏期劇場 Jan 10 '13 at 9:58
  • 2
    It will be acknowledged the next time the indexers crawl the site. This can be anything from minutes to weeks, depending on how important your site is considered. You can move things along more quickly by adding the site to webmasters at Google and Bing and getting it to reread robots.txt - you can also remove pages from the index there. – Paul Jan 10 '13 at 12:03
  • 1
    If you want to speed up the process you can submit a removal request at google.com/webmasters/tools/removals for Google and submit the URL at bing.com/toolbox/submit-site-url for Bing (yes, that's the recommended method for removing pages). – Kat Jan 10 '13 at 12:22
  • You should also add noindex to the index and other pages on the sub-domain – Anagio Feb 23 '13 at 18:11
  • How do I do when there is common root folder and subdomain names are dynamic? stackoverflow.com/questions/21229235/… – Gopi Jan 22 '14 at 6:21
6

Using a robots.txt file in your subdomain will help (and Google will obey this), but another step you can take is to specify with a Google Webmasters account that you don't want this subdomain to be indexed. You can also use a meta tag on all pages in the subdomain:

    <meta name="robots" content="noindex">

If this happens to be a site that you are only using for internal testing, limiting visibility to a specifies set of IP addresses in your virtual hosts file would further hide the site.

|improve this answer
  • In all pages????? oh no.. – 夏期劇場 Jan 10 '13 at 9:05
  • @4lvin It's done dynamically from the server side. You only need to code it once. Please don't tell me you're coding individual HTML pages. – Kenzo Jan 10 '13 at 10:36
  • LoL, actually "yes". Because this is very old/legacy web service system. (thats why it needs to be bring down) Many isolated pages inside. :D – 夏期劇場 Jan 10 '13 at 10:37
  • @4lvin Ouch...good luck with that! – Kenzo Jan 10 '13 at 10:39
  • 1
    Instead of embedding a noindex meta tag in every page, you can instead return an X-Robots-Tag HTTP response header from your server for the specified subdomain - this you only have to do once. – MrWhite Jan 10 '13 at 11:27
2

Solutions from Kenzo and Paul are good, you can put meta tags noindex on your web pages and add robots.txt to disallow robots.

But in my opinion, the best solution is to use password authentification on your sub domain. This is the only solution you're sure robots can access and index your web site. If you use Apache, you can implement htpasswd.

|improve this answer
2

...currently showing in the SEO results?

The other answers are more about proactively preventing the indexing of a (sub)domain (which is primarily what you are asking in the question) rather than actively removing your pages from the search results, which might be more what you are after, judging by your other question.

You still need to block your site in robots.txt and serve a noindex meta tag (or X-Robots-Tag HTTP response header), as stated in the other answers, but you also need to block access to your pages, returning a 404 or 410 status code.

You can read more about this on the Google Webmaster Tools help page:
Requirements for removing content

Once you have all these in place then you can use the Remove URLs tool in Google Webmaster Tools. However, this only applies to individual URLs, not an entire site, but it's a start. (Google states that using robots.txt, noindex and serving a 404 are the requirements in order to use the GWT removal tool.)

However, if you still want regular users to be able to access the site by typing the URL, then it's a problem - as your content is no longer available. You could password it, as Zistoloen suggests, however this will return a 403 (Forbidden) by default which you would need to override to return a 404/410. You could cloak your content, returning a 410 to the Googlebot and allowing everyone else to enter - but what about people clicking on results in the SERPs?

But if you want Google to remove your content in the fastest possible time then you need to remove it from the "Google" internet.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.