2

We host our sites' static content on S3. We also have a very open robots.txt;

User-agent: *
Allow: *

This is because in Webmaster tools I get thousands of warnings Sitemap contains urls which are blocked by robots.txt.

The images are listed in my sitemap along with a content item and use the correct path; http://mybucket.s3.amazon.com/image/path.jpg.

Can I add a remote URL in my robots.txt? I'm assuming, with such a liberal robots.txt on s3, that this is a restriction in my site's robots.txt.

Has anybody else stored images on s3 and put them in a sitemap?

1
  • You put a separate robots.txt on the S3. Commented Apr 29, 2016 at 9:09

1 Answer 1

1

Allow: * should actually be be Allow: /.

That could be the issue... because, really, using Allow: is somewhat meaningless by itself -- its purpose is for allowing a sub-path within a denied path. As it stands, it seems possible that your file is being misinterpreted.

Anything not denied is supposed to be implicitly allowed.

If you want to allow everything, you should instead use Disallow: with nothing after it... or of course you could just delete your /robots.txt file entirely, since a 4xx error should be interpreted by a crawler as "no restrictions here -- have fun!"

5
  • Hi Michael, thanks for the answer. We had no robots.txt originally and were getting these warnings in Webmaster Tools, so I added a robots as I assumed it was blocking by default. Can I add something in the local site robots to allow the remote file store?
    – Christian
    Commented Apr 29, 2016 at 4:13
  • Your use of the words "local" and "remote" makes me realize that I misread part of your question, actually... You said "we host our sites' static content" but I saw "we host our static site's content" ... I didn't realize the entire site wasn't a static site in S3 on a single domain name. So, yes, in that light, it sounds like this is not about S3 but about restrictions in the robots.txt file on the main site... which is not the brief one in the question. (right?) Also, no, you can't provide rules for any other origin in robots.txt. Commented Apr 29, 2016 at 4:25
  • Ok, so, currently then there should be no reason why Google could not access our static content?
    – Christian
    Commented Apr 29, 2016 at 5:56
  • 1
    Well, no, I don't see why... not unless, as I pointed out initially, Allow: * is being misinterpreted. Commented Apr 29, 2016 at 11:50
  • I updated my robots.txt to / instead of * and while nothing has been indexed, all my warnings have disappeared. Fingers crossed.
    – Christian
    Commented May 1, 2016 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.