Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Will removing robots.txt make the website 'searchable' again.
Curious if you had a robots.txt to avoid indexing, but changed your mind and wanted to get indexed.

share|improve this question

migrated from stackoverflow.com Jan 20 '11 at 22:00

This question came from our site for professional and enthusiast programmers.

5 Answers 5

Yes. It will start showing whenever the spider for the particular site next hits the site and doesn't find robots.txt. Different sites may have different intervals...I know in particular Yahoo has some set of arcane rules it follows, but it does eventually show up again.

share|improve this answer

Yes. The search engines don't save your robots.txt. You may have to wait a short while for your site to be scraped again, however.

share|improve this answer

The Robots.txt serves as a guide for search engines when they arrive at your site and begin their crawl. If you are suggesting that you've disallowed search engines from crawling your site and decide you wanted it index thereafter, you may want to expedite the process.

Googlebot (Google's crawler), for instance, engages site depending on a number of factors, including "PageRank, links to a page, and crawling constraints such as the number of parameters in a URL", which are all based on a super secret computer algorithm.

One way to expedite the process of indexation is setting up a Webmaster Tools account (for both Google and Bing), and submitting your Robots.txt (which includes permission to crawl the site) and an XML sitemap. This suggests to search engines that your site exists and is operational.

share|improve this answer

You can always use Google Webmaster Tools to request reindexing of your website. Resubmit the sitemap and Google will know that it have to check your site soon.

share|improve this answer

simply delete robots.txt and generate few backlinks so crawl comes to your site again ASAP and submit your sitemap to webmaster tools and you're done.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.