2

I have created a community driven news aggregator like Hacker News. Each news link has a comments page. Hundreds of links are added each day. I tried to create an XML sitemap for Google Search Console using an online generator. It is listing all the links (including the comments page for each link) into the XML sitemap. The main links of the site are 

home - example.com/?page=home
users- example.com/?page=users
blog - example.com/?page=blog

The contents of the home page can be filtered and corresponding links are displayed using URL parameters.

example.com/?page=home&type=1 will show the video links submitted to the site
example.com/?page=home&type=2 will show the blog links submitted to the site

like the type there is one other filters - topic
example.com/?page=home&topic=1 will display all the technology related news
example.com/?page=home&topic=2 will display all the business related news

these filters can also be used together 

example.com/?page=home&topic=1&type=1 will display all video links to technology

comment page for each news is dynamically generated when comment link on an news link is clicked.

example.com/?page=comment&newsid=144

Which will open a comment page for news with the id 144.

The problem I am having is that online generator is going to all the comment pages and including this in the sitemap. This will be thousands of comment pages.

I have decided to manually create the sitemap.

Blog pages can be navigated in the same way.

example.com/?page=blog&blogid=122

The users page displays the users info and links to the profile The profile is also a dynamic page like comments and does not exist till a userid is specified

example.com/?page=profile&userid=14

All the users profiles are also being listed by the sitemap generator.

What all links should be included in the XML sitemap?

What should be included and what should be left out?

Improve this question
1
  • Leave out duplicate pages and pages that use parameters that will largely be duplicates. For the record. I would not bother with a sitemap if your site can be properly crawled. Sitemaps or generally used to audit if a search engine can crawl your site. Sitemaps are generally useless for most sites. It has no effect on search performance and will not speed up indexing in most cases. I would skip this step and only consider a sitemap to solve a specific problem.
    – closetnoc
    Aug 10, 2017 at 17:12

1 Answer 1

Reset to default
2

For all your questions:

  • What are the essential links to be included?
  • What all links should be included in the xml sitemap?
  • What should be included and what should be left out?

reply the Search Console Help of Google - Build and submit a sitemap: "...Decide which pages on your site should be crawled by Google...". That is, in the sitemap you include the links that you wish were indexed by search engines. Pay attention to the following Google recommendations: - "Break up large sitemaps into a smaller sitemaps to prevent your server from being overloaded if Google requests your sitemap frequently. A sitemap file can't contain more than 50,000 URLs and must be no larger than 50 MB uncompressed." - "Use a sitemap index file to list all your sitemaps and submit this single file to Google rather than submitting individual sitemaps." Also pay attention to the Google's instructions for news sitesmap - Creating a Google News Sitemap: "Your News Sitemap should contain only URLs for your articles published in the last two days". Here is the key phrase YOUR ARTICLES.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.