Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

So I have created a video website which has thousands of videos and thousands of videos get added to it on a daily basis. Here is my problem :-

I have created a website which basically loads the skeleton in html and puts all the content through javascript and Ajax. The problem is search engines aren't going anywhere except for the home page.

Is there a way say in robots.txt where i give a link to a single html which has links to all these videos ?

I agree my site is not accessible for a non-javascript user but stats show that this ratio is very low ( > 0.2 % ).

Is there a way I can still keep the complete AJAX website and still get each individual videos listed on google ?

share|improve this question

You should add sitemap.xml to your site which gets updated with each new video added. Refer google's documentation for sitemaps.
About Sitemap

share|improve this answer
+1 You're presumably using some form of CMS to handle all this content. You need to work out a way for it to produce a sitemap file for search engines. – Su' Nov 21 '11 at 10:12
Ah, that should work. Thanks. Also does the sitemap.xml size have a problem ? We are expected to have 3-4 million videos by end of next year. – MarutiB Nov 21 '11 at 10:15
If it is that large amount of videos, you need not add all of them to the sitemap. Just add the recent few thousand videos. Once they get indexed there is no need to keep them in sitemap.xml. This is how even the stackoverflow maintain it's sitemap.xml file for fresh questions. – Jigar Tank Nov 21 '11 at 10:55

Google suggests This solution for indexing ajax sites which use 'hash bang' urls (I don't know if yours does).

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.