Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

Question

I've structure my video site like this:

/ (root)
 index.html
-/videos
   -/video-id1  (dynamically generated subdirectory when user uploads a video)
        /index.html  (dynamically generated file when user uploads a video)

The subdirectories and files in them are created dynamically each time a user uploads a video. I thought using a sitemap.xml would be a good choice to get those files indexed in search engines because using PHP I can edit the sitemap dynamically after each page is created. But this is quite hard to do (since it's an XML document).

So I thought of using a robots.txt instead of a sitemap to get them indexed, but would that be helpful at all for this?

In other words, I'm trying figure out how I can submit dynamically created subdirectories and files to search engines? Using Stack Exchange as another example, how do questions like this one get listed in search engines every time a user asks a questions?

I think they use sitemap.xml for every question, as can be seen here.

Answer 1

you should use both:

• crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag tells crawlers what you consider most important. Google accepts sitempas following the XML-Sitemap-Protokol 0.9 which is explained in detail at www.sitemaps.org
• crawling discouragement using a robots.txt – Providing a robots.txt you can try to tell a spider which directories and files you do not want to have crawled. This may first sound a bit strange, but it can indeed make sense to try to exclude pages like "sign up" or "subscribe to newsletter" from crawling. Here's some more information about excluding directories and files at tools.seobook.com/robots-txt/

Both techniques can help because they can be respected by search engines – but they don't have to follow your instructions listed in sitemap.xml and robots.txt.. So don't rely on either – but still do use both.

One addition: I would also recommend generating the sitemap.xml using php whenever a new file is uploaded – or whenever you actually trigger it. This gives you a) 100% control of what is listed, in what order and what tags are used and b) I consider a static file superior to one that the crawler has to render itself every time it visits.

PS: you'll probably want to use a robots.txt anyways – even if you don't want to exclude anything from crawling – since it can be so useful in many different situations…

Answer 2

I thought using a sitemap.xml would be a good choice to get those files indexed in search engines because using PHP I can edit the sitemap dynamically after each page is created. But this is quite hard to do (since it's an XML document).

It's not as difficult as you think. Using PHP, you can create the XML in a dynamic fashion and then output it with the following header:

header("Content-type: text/xml");

i.e. Select your dynamic pages from your database, create an XML structure and output them with the header above. Then, provide Google Webmasters with a link to your PHP script.

No need to edit any XML files with PHP.

Answer 3

A robots file tells search engines and others where to go and, more so, what to avoid. Sitemaps tell search engines the structure of your site and how to find, possibly, hard to discover pages and paths. In your case, it sounds like you need to set up a sitemap.

Answer 4

If you think creating an XML sitemap is too complicated, you can also create a sitemap in a plain text file. This format doesn’t offer all additional features, but it’s sufficient if you only want to list all URLs to be indexed.

See http://www.sitemaps.org/protocol.html#otherformats

UTF-8. One absolute URL per line. Nothing else.

So it might look like:

http://example.com/
http://example.com/videos
http://example.com/videos/video-id1
http://example.com/videos/video-id2
http://example.com/videos/video-id3