2

In my robots.txt for my domain that just hosts images, I used the following lines:

user-agent: *
noindex: /

I thought I was doing search engines a wonderful favor by not having them crawl through their database of URLs that are part of my site but instead, only crawl at one file to get the hint. The problem is that google has a problem.

In google webmaster tools, in the domain menu the image hosting domain has a red exclamation mark to the left of it and to the right of it, instead of seeing "No new messages or recent critical issues." I instead see "Check property health". I click on it and see "Some important page is blocked by robots.txt.". I then click on "Some important page" which directs me to my image hosting domain name itself. The thing is that link is supposed to direct visitors to the other page content domain.

I understand I could do a 301 redirect from the image domain root to the right place but that doesn't concern me. Personally I'd rather have everything on that image domain de-indexed including the root.

I double-checked the sitemap section and google states that no sitemaps have been submitted. When I check the index status, google states I have one URL indexed on that image domain.

The only idea I have in mind is to somehow modify noindex: / so that it references everything but the root folder in the domain and make the default page include a no-index directive in the HTML.

Does anyone have a better solution to this so google doesn't complain?

The idea is to have the image domain serve strictly images to the pages accessible at the content domain.

Improve this question
7
  • That's a good one! Not sure what to say to help. I will think on this for a while. You might want to mention your sub-domain structure like you did for a recent question- it may help others to understand the question better. Cheers!!
    – closetnoc
    Oct 1, 2015 at 2:30
  • I did in brief here. The image domain only hosts images and the content domain hosts the pages themselves. Inside the pages, there are URLs that point to the image domain. Yes I'm still using domain sharding. This question is more focused on the image domain. I'm just not sure why google is so upset over it. btw. Todays earnings for me is $0.00. I have a gut feeling that if there is the tiniest mistake in ANY of my google accounts then my income for the day is $0.00.
    – Mike -- No longer here
    Oct 1, 2015 at 2:35
  • I was making 5-6 most days until the 17th of June. I make something each day, but it can be as little as $.02. Before the drop, I was gaining in search traffic and had reasonable performance numbers considering. I am slowly climbing back... I think... so I know how you feel. I just want to pay the phone bill each month and my site does not suck near as bad as it used to... ;-) Still working on it even though it is discouraging.
    – closetnoc
    Oct 1, 2015 at 2:40
  • The Noindex directive in robots.txt is not documented and is flagged as an error by Google's robot.txt tester. Reference: webmasters.stackexchange.com/questions/84884/…
    – MrWhite
    Oct 1, 2015 at 11:22
  • @w3d read my G+ post and JohnMu's answer about subject: goo.gl/ge1yBB
    – Evgeniy
    Oct 1, 2015 at 15:54

1 Answer 1

Reset to default
0

I would noindex everything on this domain with X-Robots-Tag. Excluding images from crawling will not work, because images are linked from the content domain, and because Google will whine that important resources are blocked.

Your idea to redirect the main page is absolutely correct - I would doubtless do it in addition to noindexing.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.