Creating sitemap for Googlebot - how to mark dynamic content / dynamic subpages?

Question

I have a website that is an Internet forum. This forum has many categories, and a single category page that contains a lot of subpages with listed threads.

This Internet forum is brand new, and about a week ago I filled it with a few hundred thousand threads. I then looked at my Google Webmasters Tools page to see any changes in indexing, but the index went up from 300 to about 1200, so that means it did not index my added threads (although it added something).

The following is what my sitemap.xml contains, which I uploaded to their website. Of course there is a lot more code, this is just a snippet for a single category. In my real sitemap file I have all the categories listed as below:

<url>
  <loc>http://mysite.com/Forums/Physics</loc>
  <changefreq>hourly</changefreq>
</url>

Now, I would expect Googlebot to go into mysite.com/Forums/Physics, and crawl through all the subpages with thread links, and then crawl inside of each thread and index its content.

How can I achieve this? Also if this is unclear, I will add a real link to my website.

Answer 1

Just add those pages to the sitemap. Otherwise you will just have to depend on Google to follow the links it finds on each subpage within /Forums/Physics. For such a large quantity of pages you will want this to be an automated process. This includes when new pages are added. StackOverflow and other large sites do this as it would be impossible to do otherwise.

Also, just because you add a lot of pages, doesn't mean Google crawl and/or index those pages. There are many factors that determine whether content will be added to Google's index. Sometimes pages are added slowly, sometimes they are ignored because they are consider low quality.

As long as those pages are available to be crawled and they are not low quality, they should eventually be added to Google's index. There isn't much you can do to speed up the process although getting quality links to those pages certainly will help.