Linked Questions

31
votes
6answers
78k views

How to prevent a PDF file from being indexed by search engines?

I have a link to PDF document on a public web-page. How do I prevent search engines from indexing this link and PDF document? The only idea I thought of is to use CAPTCHA. However, I wonder if ...
7
votes
3answers
20k views

How to resolve Google "Indexed, though blocked by robots.txt"

Recently Google has been complaining about certain pages saying: Indexed, though blocked by robots.txt I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
5
votes
3answers
498 views

Remove large number of useless page from Google index using robots.txt

A large number of auto-generated pages from the search of my site has been indexed by Google due to an error. I'm trying to clean this in order to be sure that only the most high quality content is ...
5
votes
2answers
526 views

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google? Yesterday I ...
3
votes
2answers
2k views

URLs with 'NoIndex` in robots.txt are being indexed by Google

In my robots.txt file (http://www.tutorvista.com/robots.txt), I'm using Noindex: /content/... to disallow indexing: This should mean that http://www.tutorvista.com/content/ and anything below this ...
2
votes
2answers
1k views

Google displays search results even after removing URLs in Search Console

I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" ...
2
votes
2answers
647 views

How to tell search engines to not index entire image domain without making them waste server bandwidth or making google complain

From what I learned, there's one way I could cause all URLs on a domain strictly serving images not to be indexed and that is with the x-robots-tag HTTP header. Now I check my logs and find out that ...
2
votes
1answer
71 views

deindexing a whole domain with robots.txt created problems in google webmaster tools

In my robots.txt for my domain that just hosts images, I used the following lines: user-agent: * noindex: / I thought I was doing search engines a wonderful favor by not having them crawl through ...
3
votes
0answers
59 views

Why are pages of this sub-domain indexed by google? [closed]

If you site:news.cnblogs.com in google's search box, you will see a lot of web pages are indexed by google. But if you click on any of the search results, you will be redirected to account.cnblogs.com ...
2
votes
0answers
63 views

Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server

My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...