Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a static webpage written in HTML and CSS. I do not want it to be indexed by Google. So I made a robots.txt file in the parent directory and disallowed the URL.

I want to know:

  1. Is that the right way of doing it?
  2. How do I verify that my webpage is not being indexed by Google anymore?
share|improve this question
    
Why? Just curious. –  Martijn May 2 at 14:55
    
robots.txt file must be on root directory. sample.com/robots.txt must be exist. –  Huseyin May 2 at 16:26

2 Answers 2

Here are what you need to do:

  1. To prevent indexing of your site by Google, using a robots.txt is not enough. Additionally, you need to put a <meta name="googlebot" content="noindex"> in all your pages. All details about this are in this Google support page.

  2. To check that Google doesn't index your pages, you can do a site:www.example.com (replace www.example.com by your site URL) in Google Search and check you get no results.

If your URLs are already indexed by Google, perform the point 1., return a Gone - 410 HTTP status for all your URLs and ask for URLs removing from index in your Google Webmasters account.

share|improve this answer

Google has a help document that specifically answers this question:

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.

...

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag or x-robots-tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index. The x-robots-tag HTTP header is particularly useful if you wish to limit indexing of non-HTML files like graphics or other kinds of documents.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.