2

I have a static webpage written in HTML and CSS. I do not want it to be indexed by Google. So I made a robots.txt file in the parent directory and disallowed the URL.

I want to know:

  1. Is that the right way of doing it?
  2. How do I verify that my webpage is not being indexed by Google anymore?
2

2 Answers 2

2

Here are what you need to do:

  1. To prevent indexing of your site by Google, using a robots.txt is not enough. Additionally, you need to put a <meta name="googlebot" content="noindex"> in all your pages. All details about this are in this Google support page.

  2. To check that Google doesn't index your pages, you can do a site:www.example.com (replace www.example.com by your site URL) in Google Search and check you get no results.

If your URLs are already indexed by Google, perform the point 1., return a Gone - 410 HTTP status for all your URLs and ask for URLs removing from index in your Google Webmasters account.

1

Google has a help document that specifically answers this question:

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.

...

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag or x-robots-tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index. The x-robots-tag HTTP header is particularly useful if you wish to limit indexing of non-HTML files like graphics or other kinds of documents.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.