I am launching a medical discussion forum for sharing study materials and discussing other topics. The forum will probably grow fast and reach at least a few hundred topics.
Should every discussion topic be included in the sitemap?
asked Apr 6, 2014 at 21:09
Add a comment
|
3 Answers
Sitemaps not sitemap when dealing with high volume of content
Your first mistake would to assume that you use 'a sitemap', when dealing with sites that contain a lot of content you need to think in terms of 'sitemaps'.
Google indexes content without a sitemap
Now it's worth mentioning that Google will pretty much index any important URL without the need of a sitemap, many don't even bother with a sitemap and ensure their site has a very friendly way of finding content without having to go down to deep in the URLs to discover the new content but others believe that a sitemap really does help Google.
Sites are issued importance ratings
Every site is issued a importance rating, if you have a low rating then you get less page crawls and indexes per a visit (Google works on resources), so more important sites get longer and more regular visits, so a sitemap doesn't necessary mean indexation - if you have millions of low quality threads being created a sitemap won't necessary mean these URLS get indexed any quicker than Google just discovering them.
Sitemap Paradox
It's worth reading our very own sitemap paradox which Jeff Atword asks the very same question regarding the usage of a sitemap on the Stack Exchange network. Your need to make up your own decision what is best for your requirements.
Making a sitemap with multiple sitemaps
If you proceed to want to create multiple sitemaps then your need to familiarize yourself with the sitemap protocol.
answered Apr 6, 2014 at 21:59
Short answer. For a new site, I would not bother but I would prepare. I would just make sure your site can be crawled by spiders. Most forum software is fine for this. If per chance that you have content that cannot be reached via the crawl, then a sitemap may help. There is some exception to this of course.
Sitemaps, generally speaking, only really advantage medium to larger sites. Google, for example, will largely ignore the sitemap of a smaller site that can be crawled. When you reach say about 5,000-10,000 posts or more, then I would work on the concept of a sitemap.
My experience does not include the concept of a sites importance. I think it is in actuality a matter of sites size and complexity as to whether a sitemap is beneficial to a search engine. Bing, is said to be different. But Bing has been different from the start. Bing/MSN, Yahoo, and all the rest failed to see the importance of a scalable fresh index instead focusing on other things instead of crawling the Internet. That is why Google is where it is. Google will read a sitemap of a larger site and spider that site rather rapidly regardless of the sites current or past performance. But, as I said earlier, it may ignore a smaller sites sitemap preferring to crawl it the old fashion way. It is strange the tactic that Bing has taken on still to this day. It is odd that after all the talk about competing, and all the years of failing, they still do not get what separates Google from the rest- getting down to the business of indexing the Internet. Bing has a weird selection process that is impossible to move if they are not indexing because of the weird selection process. It is a self defeating cycle. I would not worry about Bing yet. I would worry about making Google happy. In the end, the rest will follow.
Now you said that your site is a discussion forum which is a bit harder, in some manner of speaking, to handle than say a blog. You will want each of the discussion topic pages and each user supplied question and forum of course. The way I think I would go about it is to create a sitemap file per topic you create in your forum. I would also create a sitemap for any static pages as well. Then I would create an index sitemap that includes each forum topic (again the ones you created). From there, then I would create an entry into the appropriate sitemap file for every user supplied question/discussion.
The idea is to allow room for growth. However, if your site becomes really active, you will run into limits. Each sitemap file can only have 50,000 entries. This is not a hard limit, but a recommended limit. I cap mine at 45,000 just to be safe. This means that if you approach about 45-50k entries in any forum topic you created, you will have to create an additional sitemap for that topic. For example: headtrauma.xml could expand with headtrauma2.xml then headtrauma3.xml with these new sitemap files added to the index sitemap.
If this gets confusing, it will not be if you dig in. Just start small with a plan to grow. That is what I am trying to say. Create what ever you create with volume in mind even if you are not there yet.
I do not know if your forum software has a feature for this. Just in case: you can easily roll-your-own. I wrote and deployed mine in less than 30 minutes with no growth limits! You are a smart guy. I know this! If you chose to, I am sure you can create a custom sitemap generator that perhaps can run as a cron job once a day. I do not advise getting more granular than this since Google will only likely check it once in a day though they have checked more times a day for extremely active sites.
-
I am using the Yoast SEO plugin for Wordpress for sitemap generation and it actually pings Google whenever a new article is submitted. I don't have experience in sitemaps but I trust the author of the module, so why do it myself when I can use the experience of others and spend the time saved on other pleasant things :) Apr 7, 2014 at 6:51
-
I agree. Wordpress and other software has gotten so sophisticated these days that I often do not advise recreating the wheel unless there is something that needs to be done that varies from the norm. I rolled my own bit of code that is automation oriented and not at all user friendly so I really had to write my own sitemap component. The options that existed required spidering a site that is about 500,000 pages and growing rapidly. I found that rather silly. Apr 7, 2014 at 7:10
No need to include each forum threads in Sitemap. Whenever you are creating a thread in forums means you are adding additional information to your site, Google will always look for updates where ever the updates it will find it will index. If you don't use sitemap also Google will index your site, sitemap is just an indication of how many URL's having our website. Making our site crawl engines friendly we are using Sitemap.
Thank you.
