Linked Questions
10 questions linked to/from How does "Noindex:" in robots.txt work?
3
votes
0
answers
71
views
Why are pages of this sub-domain indexed by google? [closed]
If you site:news.cnblogs.com in google's search box, you will see a lot of web pages are indexed by google. But if you click on any of the search results, you will be redirected to account.cnblogs.com ...
- 173
3
votes
0
answers
592
views
Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server
My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...
- 193
9
votes
3
answers
21k
views
How to resolve Google "Indexed, though blocked by robots.txt"
Recently Google has been complaining about certain pages saying:
Indexed, though blocked by robots.txt
I am confounded by this error. Yes the page, is blocked by robots.txt and it has always been. ...
- 473
3
votes
2
answers
2k
views
URLs with 'NoIndex` in robots.txt are being indexed by Google
In my robots.txt file (http://www.tutorvista.com/robots.txt), I'm using Noindex: /content/... to disallow indexing:
This should mean that http://www.tutorvista.com/content/ and anything below this ...
5
votes
2
answers
671
views
Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German
I found a strange search result for a resource blocked by robots.txt.
Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google?
Yesterday I ...
- 153
5
votes
3
answers
580
views
Remove large number of useless page from Google index using robots.txt
A large number of auto-generated pages from the search of my site has been indexed by Google due to an error.
I'm trying to clean this in order to be sure that only the most high quality content is ...
- 337
2
votes
2
answers
1k
views
Google displays search results even after removing URLs in Search Console
I am developing an AngualrJS application and since it is not easy to optimize it for search engines, I put it public and let Google index the page by registering a "Property" in their "Search Console" ...
- 23
2
votes
2
answers
779
views
How to tell search engines to not index entire image domain without making them waste server bandwidth or making google complain
From what I learned, there's one way I could cause all URLs on a domain strictly serving images not to be indexed and that is with the x-robots-tag HTTP header. Now I check my logs and find out that ...
- 13.6k
2
votes
1
answer
81
views
deindexing a whole domain with robots.txt created problems in google webmaster tools
In my robots.txt for my domain that just hosts images, I used the following lines:
user-agent: *
noindex: /
I thought I was doing search engines a wonderful favor by not having them crawl through ...
- 13.6k
34
votes
5
answers
90k
views
How to prevent a PDF file from being indexed by search engines?
I have a link to PDF document on a public web-page. How do I prevent search engines from indexing this link and PDF document?
The only idea I thought of is to use CAPTCHA. However, I wonder if ...
user642604