1

I've added robots.txt on my staging site at staging.mydomain.com as:

User-agent: *
Disallow: /

I then added and verified the staging site in Google Webmasters Tools. In Crawl > Blocked URLs, I can see robots.txt listed with the status as 200(Success). Further down that page, when I clicked on Test button to test http://staging.mydomain.com/, it gives me the result as:

Allowed
Detected as a directory; specific files may have different restrictions

Looks like this is the wrong result. What have I done wrong? Do I have to wait for some time to have google read robots.txt?

Within the staging site, I have other folders such as:

http://staging.mydomain.com/test1/
http://staging.mydomain.com/test2/

Obviously I want to disallow indexing all of these. When I do a test for these folders, the result shows up as Allowed. Do I need to add robots.txt within each of the sub-directories?

6
  • 1
    Have you added your robots.txt file into root directory of staging.mydomain.com, not into mydomain.com? Jun 15, 2014 at 15:18
  • It would be a good idea to protect these via HTTP Auth at least. This problem wouldn't exist in the first place.
    – ek9
    Jun 15, 2014 at 18:58
  • @Marian, robots.txt file is inside the public_html directory of staging.mydomain.com. It is accessible as staging.mydomain.com/robots.txt. Jun 15, 2014 at 23:19
  • 1
    @edvinas.me, HTTP Auth password-protects the access but will it help disallowing indexing the entire staging site in search engines? Jun 15, 2014 at 23:29
  • Indirectly. Google bot will not be able to access HTTP Auth protected websites. I would guess you also do not want anyone else accessing the staging site without specifically being allowed to.
    – ek9
    Jun 16, 2014 at 7:31

1 Answer 1

2

Looks like what done is perfectively fine.

A typical robots.txt for a production site might be as simple as:

User-agent: *
Disallow:

This is the least restrictive. It says that all crawlers are allowed to crawl the entire site.

For our dev or staging site, we want to use the following:

User-agent: *
Disallow: /

This requests that the entire site not be crawled.

But some time if you didn't take proper precautions prior to creating your dev or staging sites, there's a good chance that the search engines found your work-in-progress.

What now? Well, let's be careful here.

1.First, understand that search engines will cache your site for a certain length of time.

2.Second, you'll need to keep in mind that restricting crawling of your site does not mean that existing indexed pages will disappear from search engine results.

If you find your staging site pages in search results, it's a good idea to go ahead and tell search engines not to index each page. The best way to is to add a "noindex" meta tag to all your pages. The noindex tag looks like this:

<meta name="robots" content="noindex" />

OR :

Advised Approach:

1.Add Authentication (HTTP or otherwise) infront of requests.

2.Respond with appropriate response code if not permitted (e.g. 401 Unauthorized).

3.Everything else in the Basic Approach above.

By adding a robots.txt it prevents search engines from accessing and indexing the content. However, that doesn't mean they won't index the URL. If a search engine knows about a given URL, it may add it to the search result index. You'll sometimes see these in the search results. The title tends to be the URL with not description. To prevent this from happening, the search engines need to be told not to show the content or URLs. By adding Authentication infront and not responding with a 200 OK status code it is a strong signal to the engines not to add these URLs to their index. From my experience I haven't ever seen a 401 response code page listed in a search engine index.

2
  • But the webmaster tools' crawler will not show "Allowed" even is the URL is crawled, but disallowed by robots.txt. Jun 16, 2014 at 11:05
  • Note that Sudheera is correct: robotstxt.org/faq/prevent.html — what you have should block GoogleBot. Assuming you can see your robots.txt content when you go to that very page on your website. Oct 27, 2015 at 2:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.