Tag Info

Hot answers tagged

4

So, the solution seems to be that Amazon cloudfront also evaluates my robots.txt and somehow uses different syntax rules from google. The working version of my robots.txt is the following: User-agent: Googlebot-Image Disallow: / User-agent: * Disallow: /homepage Disallow: /uncategorized Disallow: /page Disallow: /category Disallow: /author Disallow: /feed ...


1

Yes. You can. I do not know the tool you are using, but here are some clues. Not a search engine (duh!). High number of requests over a longer period of time- more than a user. Does not request an image, CSS, or JS (some do, most don't). Changes agent name and OS over a longer period of time. Comes from a web-hosting IP address block/range. Check for ...


1

Create a robots.txt in a bucket. Create another origin for your cloudfront distribution. Set your bucket's priority higher then your website. Invalidate your site's robots.txt on Cloudfront. After doing the above, Google will read the sites robots.txt when crawling your site and will get to see the different robots.txt when following links from your cdn. ...


1

Found out the problem: The CloudFront reads the robots.txt and prevents serving the content, but it parses some how different from what robots should, I guess. For instance, the following content on robots.txt: Disallow: */wp-contents/ Allow: */wp-contents/themes/ When Googlebot gets it itself, it indexes it; When CloudFront reads it, it doesn't consider ...


1

Google does not block external resources from being indexed via using a robots.txt in the root of the main site. Using a sub domain, a cdn or other is classed as an external domain therefor the only way to block the content is using a header response on the file served by the CDN itself, or by using a robots.txt on the cdn or sub domain. Using: #Google ...



Only top voted, non community-wiki answers of a minimum length are eligible