Will removing robots.txt make the website 'searchable' again.
Curious if you had a robots.txt to avoid indexing, but changed your mind and wanted to get indexed.

5 Answers 5


Yes. The search engines don't permanently save your robots.txt file. However, you may have to wait a short while for your site to be crawled again.


Yes. It will start showing whenever the spider for the particular site next hits the site and doesn't find robots.txt. Different sites may have different intervals...I know in particular Yahoo has some set of arcane rules it follows, but it does eventually show up again.


The Robots.txt serves as a guide for search engines when they arrive at your site and begin their crawl. If you are suggesting that you've disallowed search engines from crawling your site and decide you wanted it index thereafter, you may want to expedite the process.

Googlebot (Google's crawler), for instance, engages site depending on a number of factors, including "PageRank, links to a page, and crawling constraints such as the number of parameters in a URL", which are all based on a super secret computer algorithm.

One way to expedite the process of indexation is setting up a Webmaster Tools account (for both Google and Bing), and submitting your Robots.txt (which includes permission to crawl the site) and an XML sitemap. This suggests to search engines that your site exists and is operational.


You can always use Google Webmaster Tools to request reindexing of your website. Resubmit the sitemap and Google will know that it have to check your site soon.


simply delete robots.txt and generate few backlinks so crawl comes to your site again ASAP and submit your sitemap to webmaster tools and you're done.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.