I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt

Let's take an example, I need to get this page:


According to http://www.aliexpress.com/robots.txt:

# file: robots.txt,v 1.0 2002/09/23 created by Tsing Kong
# www.alibaba.com
# <URL:http://www.robotstxt.org/wc/exclusion.html#robotstxt>
# Format is:
#       User-agent: <name of spider>
#       Disallow: <nothing> | <path>
# -----------------------------------------------------------------------------
User-agent: *
Disallow: /search/
Disallow: /productdetail/

User-agent: Baiduspider
Disallow: /promotion/
Disallow: /wholesale?

User-agent: baiduspider
Disallow: /promotion/

the previous web page URL is allowed and I could set any user agent. But that doesn't work.

Also I've tried the following but without luck:

  1. I set a user agent for chrome:

    curl_setopt ( $curl, CURLOPT_USERAGENT,
        'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'
  2. set cookies:

    curl_setopt ( $curl, CURLOPT_HTTPHEADER, array (
        Cookie: NAME1=OPAQUE_STRING1; NAME2=OPAQUE_STRING2ali_apach
    ) );

    Also I save the cookies for the request in the database, to send it back with the next request.

  3. I've changed the IP multiple times. I am using amazon ec2 (elasticbeanstalk) to do so.

What am I doing wrong ?

1 Answer 1


Here is a start:

  • Websites do not like being spidered and having their resources wasted.
  • You are using Amazon which has an absolutely terrible reputation for abusive activities.
  • You are emulating a browser and being dishonest about who you are.
  • You are clearly a bot and not providing a agent name to block access via an informational URL within the agent ID.
  • You may be accessing the site excessively.
  • You are assuming that you have the right to access the site and not assuming that you need to behave nicely.

Expect to be blocked.

  • thank you for your answer, but please note i have not sent more than 100 request, currently i am just trying. Commented Feb 22, 2016 at 22:26
  • what do you mean with amazon has terrible reputation, what does that matter . they cant insert all amazon ips into blacklist Commented Feb 22, 2016 at 22:28
  • could you please give an example of, how the user agent should be ? Commented Feb 22, 2016 at 22:30
  • @adamkaplan You are using a browser agent when clearly spidering. I can know this on my site within just a handful of accesses. As well, Amazon is the #1 abusive network there is for content scraping. You are not giving any site an opt-out. You are likely hitting the site in a way that is clearly not human. I can go on and on. I have systems that will shut you down automatically very quickly (within just a few seconds) just based upon what I see already.
    – closetnoc
    Commented Feb 22, 2016 at 22:32
  • 1
    thank you again,but i swear i did not send more than 100 requests ! please tell me what does the search engine do ? Commented Feb 22, 2016 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.