Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This question already has an answer here:

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?

share|improve this question

marked as duplicate by DisgruntledGoat, John Conde Jun 14 '13 at 11:38

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

4 Answers 4

No, neither is required.

By default your site will be indexed by Google. The robots.txt file is useful for preventing it from accessing specific directories or files. It's not a security mechanism, however, and if you don't want the public to access those pages/files, you should block access another way, such as an .htaccess file.

The sitemap.xml is also not required, but it does help Google to find all of your content. If all of your pages are cross-linked, that won't matter. Sitemaps also contain page ranking that you can suggest to Google, effectively letting the search engine know what you think is the most important content on your site.

share|improve this answer

Of course not. These two files are not required all the time.

In Google Webmaster Tools, Google says that a robots.txt is not necessary if you want that all webpages being crawled. If you want to put only these following lines in a robots.txt:

User-agent: *
Allow: /

Just don't create a robots.txt.

Moreover, a sitemap.xml helps indexing of your website but it's not obliged to use it. Googlebot is smart enough to follow links on the web and when it finds a link to one of your webpage, it follows it and crawls your webpage to index it. For example, if you have many backlinks, Googlebot will crawl often your webpages and you don't need to create a sitemap.xml.

share|improve this answer

I think in order for your page to get index by search engines like Google, Bing etc. you need to have both robots.txt and sitemap.xml files.

A quick Google shows that your robots.txt file has to be placed on root directory for your website. This link should give you more details about how-to create that file and what to do with it.

And for sitemap.xml, there are many websites that will let your generate it for free. This Google webmaster help should give you more details about how Google understands it. And how to create it.

I am using WordPress so I know that there are already plugins available that does it for you. Not sure about other blogging platforms, but they too should have it.

share|improve this answer
2  
"to get index by search engines like Google, Bing etc you need to have both Robots.txt and Sitemap.xml files" this is just plain wrong. –  Qmal Jun 13 '13 at 14:09
    
100% agree with Qmal. –  Zistoloen Jun 13 '13 at 14:41

The only thing that you need for Google to follow every link is to let Google know that your site exists. This can happen "naturally" if your site is linked from sites that already exist in Google index, alternatively you can use their site submission form to explicitly add your site to the list of sites to index here.

robots.txt and sitemap.xml are not needed, but sitemap.xml will help guide the bots through your site.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.