Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I have a static webpage written in HTML and CSS. I do not want it to be indexed by Google. So I made a robots.txt file in the parent directory and disallowed the URL.
I want to know:
Here are what you need to do:
To prevent indexing of your site by Google, using a robots.txt is not enough. Additionally, you need to put a <meta name="googlebot" content="noindex"> in all your pages. All details about this are in this Google support page.
To check that Google doesn't index your pages, you can do a site:www.example.com (replace www.example.com by your site URL) in Google Search and check you get no results.
If your URLs are already indexed by Google, perform the point 1., return a Gone - 410 HTTP status for all your URLs and ask for URLs removing from index in your Google Webmasters account.
Google has a help document that specifically answers this question:
While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.
...
To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag or x-robots-tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index. The x-robots-tag HTTP header is particularly useful if you wish to limit indexing of non-HTML files like graphics or other kinds of documents.
