Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Today I was looking at what googles shows in relation to my site and I discovered something which I wasn't expecting.

Here is what my site map looks like.

?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

So to get an idea at /pub I have a number of articles which are doFollow (checked that) and everyone of the articles has a canonical link in the header

   <link rel="canonical" href="http://domain.com/pub/article/327" />

Now I've discovered that google indexed only the /pub directory, without following the links to the articles. I read somewhere that it is not a requirement to have every article's link in the sitemap. In my case it is user generated content so I can't be updating that on a weekly basis it has to be automated in some way.

What am I doing wrong? I need to get that content indexed.

share|improve this question
up vote 0 down vote accepted

Much of the point of having a sitemap is letting Google know about new content. If you are only putting one central page in the sitemap and expecting Google to crawl from there, there isn't much point in having that sitemap -- Google can find central pages and crawl from those without much help.

I have sites with user generated content that is constantly being created. I re-generate the sitemaps nightly.

share|improve this answer
well, my idea was to actually let google know about pages which are hidden from the landing page. For instance right now my landing page is blocking the rest of the content due to the fact that I'm trying to get users to register first. I was expecting that once I get the central parts google can start from there and crawl the rest on its own. How can I get around this? Just post a couple of links on the index page and forget about the sitemap, or generate sitemaps everyday? I don't think such routines are in the style of the 21st century. – Ando Aug 20 '13 at 20:00
If your pages are not linked from your front page (or any page with pagerank), Google will probably choose not to index them, especially if they are not in your sitemap. – Stephen Ostermiller Aug 20 '13 at 20:09
I guess I'll just get rid of the sitemap and add a link to my public articles. – Ando Aug 20 '13 at 21:20
In my experience Google will index URLs that are in an xml sitemap, even if they aren't linked elsewhere on the site. See this all the time with sites that have automated xml sitemaps. – Max Aug 21 '13 at 2:15

Don't get rid of the sitemap. Just add the links to your public articles to it. The sitemap can help get your articles indexed quicker and help Google, Bing and others to understand your site better.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.