I've structure my video site like this:
/ (root)
index.html
-/videos
-/video-id1 (dynamically generated subdirectory when user uploads a video)
/index.html (dynamically generated file when user uploads a video)
The subdirectories and files in them are created dynamically each time a user uploads a video. I thought using a sitemap.xml
would be a good choice to get those files indexed in search engines because using PHP I can edit the sitemap dynamically after each page is created. But this is quite hard to do (since it's an XML document).
So I thought of using a robots.txt
instead of a sitemap to get them indexed, but would that be helpful at all for this?
In other words, I'm trying figure out how I can submit dynamically created subdirectories and files to search engines? Using Stack Exchange as another example, how do questions like this one get listed in search engines every time a user asks a questions?
I think they use sitemap.xml
for every question, as can be seen here.