Linked Questions

34 votes
5 answers

How to prevent a PDF file from being indexed by search engines?

I have a link to PDF document on a public web-page. How do I prevent search engines from indexing this link and PDF document? The only idea I thought of is to use CAPTCHA. However, I wonder if ...
user avatar
10 votes
3 answers

How to resolve Google "Indexed, though blocked by robots.txt"

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
Eaten by a Grue's user avatar
3 votes
2 answers

URLs with 'NoIndex` in robots.txt are being indexed by Google

In my robots.txt file (, I'm using Noindex: /content/... to disallow indexing: This should mean that and anything below this ...
Goutham Jabez GJ's user avatar
5 votes
2 answers

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google? Yesterday I ...
Mark's user avatar
  • 153
5 votes
3 answers

Remove large number of useless page from Google index using robots.txt

A large number of auto-generated pages from the search of my site has been indexed by Google due to an error. I'm trying to clean this in order to be sure that only the most high quality content is ...
Jonathan Meyer's user avatar
2 votes
2 answers

How to tell search engines to not index entire image domain without making them waste server bandwidth or making google complain

From what I learned, there's one way I could cause all URLs on a domain strictly serving images not to be indexed and that is with the x-robots-tag HTTP header. Now I check my logs and find out that ...
Mike -- No longer here's user avatar
2 votes
2 answers

Google displays search results even after removing URLs in Search Console

I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" ...
hans's user avatar
  • 23
3 votes
0 answers

Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server

My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...
FiddlingAway's user avatar
2 votes
1 answer

deindexing a whole domain with robots.txt created problems in google webmaster tools

In my robots.txt for my domain that just hosts images, I used the following lines: user-agent: * noindex: / I thought I was doing search engines a wonderful favor by not having them crawl through ...
Mike -- No longer here's user avatar
3 votes
0 answers

Why are pages of this sub-domain indexed by google? [closed]

If you in google's search box, you will see a lot of web pages are indexed by google. But if you click on any of the search results, you will be redirected to ...
William's user avatar
  • 173