1

Today I was looking at what googles shows in relation to my site and I discovered something which I wasn't expecting.

Here is what my site map looks like.

?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
  <loc>http://domain.com/pub</loc>
  <lastmod>2013-06-06</lastmod>
  <changefreq>weekly</changefreq>
</url>
</urlset>

So to get an idea at /pub I have a number of articles which are doFollow (checked that) and everyone of the articles has a canonical link in the header

   <link rel="canonical" href="http://domain.com/pub/article/327" />

Now I've discovered that google indexed only the /pub directory, without following the links to the articles. I read somewhere that it is not a requirement to have every article's link in the sitemap. In my case it is user generated content so I can't be updating that on a weekly basis it has to be automated in some way.

What am I doing wrong? I need to get that content indexed.

Improve this question

2 Answers 2

Reset to default
0

Much of the point of having a sitemap is letting Google know about new content. If you are only putting one central page in the sitemap and expecting Google to crawl from there, there isn't much point in having that sitemap -- Google can find central pages and crawl from those without much help.

I have sites with user generated content that is constantly being created. I re-generate the sitemaps nightly.

Improve this answer
4
  • well, my idea was to actually let google know about pages which are hidden from the landing page. For instance right now my landing page is blocking the rest of the content due to the fact that I'm trying to get users to register first. I was expecting that once I get the central parts google can start from there and crawl the rest on its own. How can I get around this? Just post a couple of links on the index page and forget about the sitemap, or generate sitemaps everyday? I don't think such routines are in the style of the 21st century.
    – Ando
    Aug 20, 2013 at 20:00
  • If your pages are not linked from your front page (or any page with pagerank), Google will probably choose not to index them, especially if they are not in your sitemap.
    – Stephen Ostermiller ♦
    Aug 20, 2013 at 20:09
  • I guess I'll just get rid of the sitemap and add a link to my public articles.
    – Ando
    Aug 20, 2013 at 21:20
  • In my experience Google will index URLs that are in an xml sitemap, even if they aren't linked elsewhere on the site. See this all the time with sites that have automated xml sitemaps.
    – Max
    Aug 21, 2013 at 2:15
0

Don't get rid of the sitemap. Just add the links to your public articles to it. The sitemap can help get your articles indexed quicker and help Google, Bing and others to understand your site better.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.