0

I would like to prevent Amazon from scraping product data on my website. So I found this document: https://developer.amazon.com/amazonbot

And this example:

User-agent: Amazonbot             # Amazon's user agent
Disallow: /do-not-crawl/             # disallow this directory

So, if I add:

User-agent: Amazonbot             # Amazon's user agent
Disallow: /                       # disallow access to all the website

or maybe

User-agent: Amazonbot             # Amazon's user agent
Disallow: /Technology/            # disallow access to Technology category page

In particular, would this prevent access to all products on the Technology page on the website?

What concerns me also is the mention of crawl delay in their 'Help' Page?

I currently have:

User-agent: *
Disallow: /admin/
Disallow: /api/
Crawl-delay: 1

User-agent: Amazonbot 
Disallow: / 

Which obviously has a crawl delay and this comment within their 'Help' page:

Today, AmazonBot does not support the crawl-delay directive in robots.txt and robots meta tags on HTML pages such as “nofollow” and "noindex".

2
  • 2
    If you don#t trust Amazon, why would you trust them to respect your robots.txt? Or even to not lie about their user-agent? It's not quite clear what you are trying to do, but one easy step (that's also easy to circumvent) is to just block all requests with a user agent string that contains amazonbot on your webserver. Commented Apr 25 at 10:08
  • Griass Di Joachim. Indeed. I agree with you why would I trust them? Perhaps I live in vain hope that they are perhaps not as bad as I believe. I currently have:- User-agent: Amazonbot Disallow: / Should I also perhaps include lowercase amazonbot as you suggest?
    – Rhothgar
    Commented Apr 25 at 10:29

1 Answer 1

0

Setting a robots.txt policy does not "prevent" any crawler from indexing your website. It is more like politely asking not to do it. The robot may respect your request or ignore it.

Since Amazon explicitly stated that they do respect robots.txt but don't support crawl-delay I would expect them to do just that, although you can block their crawler in different ways, using data they provided on their website:

  1. Identification by User-Agent

You may check for user agents containing the string "Amazonbot" (or simply "amazon") and give them a 403 error. There are multiple ways to do this, you may add the following to your VHost file if you use Apache:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} amazonbot [NC]
RewriteRule . - [R=403,L]

You may also use your firewall to perform this function. If you use CloudFlare, they provide a built-in function to block specific UAs. You may also use iptables for this, however, this would also take a toll on performance since every request would have to be inspected. Nevertheless, if you want to set it up, take a look at iptables -m string -h.

  1. Identification by IP-Address and reverse DNS

This would be a little more involved, but in general you may take an IP address of a bot accessing your website, run a reverse DNS lookup on that address and then check if the domain contains "amazon". I would certainly not suggest this as a way to apply with every request to your website since it would take a heavy performance toll, however, you might retroactively scan your log files with such a script to look for disobedient Amazonbot, that don't respect robots.txt.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.