Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I currently building a server to host many sites. How do I get google to crawl each site without submitting a site map to google for each site?

share|improve this question

migrated from stackoverflow.com Nov 9 '11 at 2:41

This question came from our site for professional and enthusiast programmers.

up vote 6 down vote accepted

How do I get google to crawl each site without submitting a site map to google for each site?

There is no definitive way to do this without you making an effort to have those sites found by Google or any other search engine. Search engines typically only crawl and index sites (actually, those sites' pages):

  1. they are told about by the webmaster (i.e. XML sitemaps, Google Webmaster Tools, etc)

  2. they find by following a link on a web page already in their index

So a web site being on the same server as another website isn't enough for that website to be found. So, if you want to make getting those sites found by Google as easy as possible create a webpage on a website already crawled and indexed by Google and link to the home page of each site. Once those pages are found Google and the other search engines will find and crawl inner pages by following links on the home page.

share|improve this answer

Google will crawl any website it finds, as long as it doesn't consider it offensive, or breaking its policies. So, you don't have to manually submit all your websites to Google. Just submit one page that will act as a directory, and update that page with a link to the front page of each new website on your server.

Submit this page to Google, it will crawl it from time to time and go to all new links contained within. Once on a new domain, Googlebot will crawl all links it finds within or going out of that domain as well.

share|improve this answer
  1. Submit the first site to Google
  2. Make sure you have links on your first site to all the other sites on your server.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.