Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm looking for the best practice to let Google know about a vast number of dynamically created content. Let's say (I mean - dream) that I'm Facebook, and I want to let Google to index all the users' posts.

  1. Sitemap.xml may be the answer for this but they are limited to 50,000 URLs in each site map. I know that I can create 500 sitemaps and create a sitemap for sitemaps, but they are also limited, 25,000,000 URLS sounds quite enough at the moment, but could cause problems in the future. I.E - Stack Overflow already has 3 Million posts, probably sitemap is not the solution for them.

  2. Creating a page with paging, and links to all the dynamic data. i guess this is what Stack Overflow did by creating this page here: http://stackoverflow.com/questions

So I think that Option 2 is the answer, but it seems to me that sitemaps might have some added value. So what should I do?

share|improve this question
    
Are there posts that aren't not going to be linked to from any other page? –  paulmorriss Jun 21 '12 at 10:43
1  
[this would be a comment, but I cannot comment yet] See what stackoverflow thinks: webmasters.stackexchange.com/questions/4803/the-sitemap-paradox –  milo5b Jun 21 '12 at 10:52
    
paulmorriss - the posts are serverd via ajax, so google isn't able to crawl them, unless i will make some special steps to do so. so I thought that by creating a sitemap with all the links to the posts, it will make my website crawlable. I guess i was wrong, after I read milo5b excelent link: "The Sitemap file isn't meant to "fix" crawlability issues. If your site can't be crawled, fix that first". –  Yaniv Jun 21 '12 at 13:50

1 Answer 1

Both options are what you should be striving for. You want to make it easy for your users and search engines to find your content. As for the XML sitemap, if your goal is to become as large as Facebook then you better be prepared to deal with things on a large scale and that includes sitemaps.

share|improve this answer
    
my users will see my content that served to them nicely via AJAX, it's only the Search Engines that i'm concerned of. what do you mean by "you better be prepared", by what? by doing a 500 sitemaps for 50,000 urls each? maybe i should create a site map and create my AJAX website crawlable? I can think about different way rather than AJAX to do it. –  Yaniv Jun 21 '12 at 13:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.