I currently building a server to host many sites. How do I get google to crawl each site without submitting a site map to google for each site?
migrated from stackoverflow.com Nov 9 '11 at 2:41
There is no definitive way to do this without you making an effort to have those sites found by Google or any other search engine. Search engines typically only crawl and index sites (actually, those sites' pages):
So a web site being on the same server as another website isn't enough for that website to be found. So, if you want to make getting those sites found by Google as easy as possible create a webpage on a website already crawled and indexed by Google and link to the home page of each site. Once those pages are found Google and the other search engines will find and crawl inner pages by following links on the home page. |
|||
|
|
|
Google will crawl any website it finds, as long as it doesn't consider it offensive, or breaking its policies. So, you don't have to manually submit all your websites to Google. Just submit one page that will act as a directory, and update that page with a link to the front page of each new website on your server. Submit this page to Google, it will crawl it from time to time and go to all new links contained within. Once on a new domain, Googlebot will crawl all links it finds within or going out of that domain as well. |
|||
|
|
|
|||
|
|