I have recently launched a website which is using a free sitemap generator. It's a large travel agency site with predominantly dynamically generated content - users can add their products as and when ...
I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: ...
I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
I've got a blog that's been running for about a year, with about 200 posts, and there should be about 220 pages to index (additional pages for about / contact etc). When I go to crawl the site I get ...
How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?