4

I am running a website but for some days I have stopped it and put the site under-construction because the Index of admin page is visible to the outside world through the Google search.

One of my friends told me that your website's index is visible and it's one step away to access the password file and he showed me that simply using the Google search.

How can I prevent this? I reported this to my hosting company but they simply replied to me

Still it's secure so you no need to worry...

Should I really don't need to worry and continue my site with the visible index of admin page?

1
  • The fact that your page is 'visible' is only because google cache the information. My friend has a 'under construction' on his page before it was setup. And he said that for 2 months later his page appeared in google as 'under construction', which really annoyed him. Commented Mar 24, 2012 at 16:49

5 Answers 5

8

There are several ways to block your admin. The easiest way would be to use htaccess to send an HTTP header to tell the search engines not to index any of those pages. Put the following code in an htaccess file in your admin directory:

Header: "X-robots-tag: noindex"

This will apply this header to every page in that directory and subdirectory below it and prevent those pages from being indexed.

You can read more about this in Google's Webmaster Blog.

2
  • i am on configure i will tell the result about it thanks for your answer
    – Krish
    Commented Mar 26, 2012 at 7:10
  • This is the best answer. Besides tediously manually removing pages using Google Webmaster Tools. Once it's indexed this will make it removed. For a protected resource I would do X-Robots-Tag: noindex, nofollow. See the dev docs Commented Oct 26, 2015 at 16:41
6

Google webmaster tools allows you to remove pages from their search index for sites that you own.

You can then use robots.txt to ask most major search engines not to index parts of your site Google cover how to create a robots file in some detail and webmaster tools has a useful generator. Example code here

Block a page:-

Disallow: /private_file.html

Block a directory

Disallow: /directory/ 
1
  • what if when i don not have that kind of set up..instead of file i handle admin panel through url. Can i block the url ??
    – Kishan Oza
    Commented Nov 16, 2017 at 12:37
3

robots.txt doesn't block Google or others from indexing URLs, it simply asks please do not crawl these URL's while you're here http://www.mattcutts.com/blog/robots-txt-remove-url/ if these blocked pages are linked to from other pages on the web Google will still index the page, noindex should be used as well as blocking with robots.txt

1
  • If a URL is blocked by robots.txt Googlebot can't crawl it to see if it has noindex. If you use noindex you should use it instead of disallowing in robots.txt Commented Apr 27, 2023 at 16:48
3

Google and other search engines can easily crawl and index your WordPress theme files and scripts. It might reduce your page rank and SEO value. To fix this, create or open a robots.txt file in the root of WordPress and add the following code:

User-agent: *
Disallow:  /wp-admin/
Disallow:  /wp-includes/
Disallow:  /wp-content/themes/
Diasllow:  /wp-content/plugins/
2
  • You're assuming the site is WordPress, which it doesn't sound to be. Commented Oct 26, 2015 at 10:30
  • @AndrewLott this answer can be easily adopted to any other site. Just change the links and place the robots.txt in the root directory. Commented Sep 19, 2018 at 10:05
2

robots.txt is very dangerous. When hackers find your robots.txt, they will see all the forbidden files and directories and will find a way to access them. Use .htaccess on each folder and state the Options -Indexes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.