Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?
I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically generated subdirectory when user uploads a video) /index.html (dynamically generated file ...
I've got a blog that's been running for about a year, with about 200 posts, and there should be about 220 pages to index (additional pages for about / contact etc). When I go to crawl the site I get ...
I'm looking for the best practice to let Google know about a vast number of dynamically created content. Let's say (I mean - dream) that I'm Facebook, and I want to let Google to index all the users' ...