I am running a website but for some days i stopped it and put the under-construction page because the Index of admin page is visible to the outside world through the Google search. One of my friend told me that your websites index is visible and its one step away to access the password file and he shows me that very simply using the Google search.

How can i prevent this and i am hosting my site with a hosting company and i report about this to them but they simply replied to me still its secure so you no need to worry...

am i really don need to worry and continue my site with the visible index of admin page?

  • The fact that your page is 'visible' is only because google cache the information. My friend has a 'under construction' on his page before it was setup. And he said that for 2 months later his page appeared in google as 'under construction', which really annoyed him. – Ramzi Kahil Mar 24 '12 at 16:49

Google webmaster tools allows you to remove pages from their search index for sites that you own.

You can then use robots.txt to ask most major search engines not to index parts of your site Google cover how to create a robots file in some detail and webmaster tools has a useful generator. Example code here

Block a page:-

Disallow: /private_file.html

Block a directory

Disallow: /directory/ 
  • what if when i don not have that kind of set up..instead of file i handle admin panel through url. Can i block the url ?? – Kishan Oza Nov 16 '17 at 12:37

There are several ways to block your admin. The easiest way would be to use htaccess to send an HTTP header to tell the search engines not to index any of those pages. Put the following code in an htaccess file in your admin directory:

Header: "X-robots-tag: noindex"

This will apply this header to every page in that directory and subdirectory below it and prevent those pages from being indexed.

You can read more about this in Google's Webmaster Blog.

  • i am on configure i will tell the result about it thanks for your answer – Krish Mar 26 '12 at 7:10
  • This is the best answer. Besides tediously manually removing pages using Google Webmaster Tools. Once it's indexed this will make it removed. For a protected resource I would do X-Robots-Tag: noindex, nofollow. See the dev docs – Anthony Hatzopoulos Oct 26 '15 at 16:41

robots.txt doesn't block Google or others from indexing URLs, it simply asks please do not crawl these URL's while you're here http://www.mattcutts.com/blog/robots-txt-remove-url/ if these blocked pages are linked to from other pages on the web Google will still index the page, noindex should be used as well as blocking with robots.txt


Google and other search engines can easily crawl and index your WordPress theme files and scripts. It might reduce your page rank and SEO value. To fix this, create or open a robots.txt file in the root of WordPress and add the following code:

User-agent: *
Disallow:  /wp-admin/
Disallow:  /wp-includes/
Disallow:  /wp-content/themes/
Diasllow:  /wp-content/plugins/
  • You're assuming the site is WordPress, which it doesn't sound to be. – Andrew Lott Oct 26 '15 at 10:30
  • @AndrewLott this answer can be easily adopted to any other site. Just change the links and place the robots.txt in the root directory. – Abdul Sadik Yalcin Sep 19 '18 at 10:05

robots.txt is very dangerous. When hackers find your robots.txt, they will see all the forbidden files and directories and will find a way to access them. Use .htaccess on each folder and state the Options -Indexes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.