title: PHP Curl request blocked but works in browser
body: <p>I have a script that does some web scraping from a website. That website requires an OPTIONS request first and then a POST request that results in a json returned with the data I need. It was working fine, I ran it only like 6 or 7 times, the results were ok, but now my POST request results in a HTML page that says <em>"Your access to our site is blocked. Our site is using a security service to protect against attacks. Your access is triggering one of our security measures."</em> </p>
  
  <p>I know they haven't blocked my IP because I can access the same site through the browser using the same IP and it works fine. So how do they know I'm using curl and not a real browser?</p>
  
  <p>As you can see in my code below, I've changed the user-agent to match the one in my browser, I'm sending the exact same headers my browser sends.</p>
  
  <pre><code>$useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36';
  
  $headerOptions = array(
      ':authority: flight-pricing.maxmilhas.com.br',
      ':method: OPTIONS',
      ':path: /search?time='.$epochtime,
      ':scheme: https',
      'accept: */*',
      'accept-encoding: gzip, deflate, br',
      'accept-language: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7',
      'access-control-request-headers: authorization,content-type',
      'access-control-request-method: POST',
      'origin: https://www.maxmilhas.com.br',
      'user-agent: '.$useragent
  );
  
  $url = 'https://flight-pricing.maxmilhas.com.br/search?time='.$epochtime;
  
  $curl = curl_init();
      curl_setopt($curl, CURLOPT_USERAGENT, $useragent );
      curl_setopt($curl, CURLOPT_AUTOREFERER, true);
      curl_setopt($curl, CURLINFO_HEADER_OUT, true);
      curl_setopt($curl, CURLOPT_COOKIEJAR, "C:/wamp64/www/maxmilhas/tmp/cookieoptions.txt");
      curl_setopt_array($curl, array(
            CURLOPT_URL =&gt; $url,
            CURLOPT_RETURNTRANSFER =&gt; true,
            CURLOPT_ENCODING =&gt; "",
            CURLOPT_MAXREDIRS =&gt; 10,
            CURLOPT_TIMEOUT =&gt; 30,
            CURLOPT_CUSTOMREQUEST =&gt; "OPTIONS",
            CURLOPT_HTTPHEADER =&gt; $headerOptions  
      ));
  
      $response = curl_exec($curl);
      $err = curl_errno($curl);
      file_put_contents('log/headerout/options.txt',curl_getinfo($curl, CURLINFO_HEADER_OUT ));
  
  curl_close($curl);
  
  if ($err) {
      echo "cURL -OPTIONS Request- Error #:" . $err;
  }
  
  file_put_contents('log/options.txt',$response);
  
  $headerPost = array(
      ':authority: flight-pricing.maxmilhas.com.br',
      ':method: POST',
      ':path: /search?time='.$epochtime,
      ':scheme: https',
      'accept: application/json, text/plain, */*',
      'accept-encoding: gzip, deflate, br',
      'accept-language: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7',
      'authorization: '.$authorization,
      'content-length: 145',
      'content-type: application/json;charset=UTF-8',
      'origin: https://www.maxmilhas.com.br',
      'referer: '.$refererPost,
      'user-agent: '.$useragent
  );
  
  $curl2 = curl_init();
      curl_setopt($curl2, CURLOPT_URL, $url);
      curl_setopt($curl2, CURLOPT_USERAGENT, $useragent );
      curl_setopt($curl2, CURLOPT_REFERER, $refererPost);
      curl_setopt($curl2, CURLINFO_HEADER_OUT, true);
      curl_setopt($curl2, CURLOPT_AUTOREFERER, true);
      curl_setopt($curl2, CURLOPT_COOKIEJAR, "C:/wamp64/www/maxmilhas/tmp/cookiepost.txt");
      curl_setopt($curl2, CURLOPT_POSTFIELDS, $payload);
      curl_setopt_array($curl2, array(
          CURLOPT_RETURNTRANSFER =&gt; true,
          CURLOPT_ENCODING =&gt; "",
          CURLOPT_MAXREDIRS =&gt; 10,
          CURLOPT_TIMEOUT =&gt; 30,
          CURLOPT_CUSTOMREQUEST =&gt; "POST",
          CURLOPT_HTTPHEADER =&gt; $headerPost
      ));
  
      $response2 = curl_exec($curl2);
      $err2 = curl_errno($curl2);
      file_put_contents('log/headerout/post.txt',curl_getinfo($curl2, CURLINFO_HEADER_OUT ));
  
  curl_close($curl2);
  
  if ($err2) {
      echo "cURL -POST Request- Error #:" . $err2;
  }
  
  file_put_contents('log/post.txt',$response2);
  </code></pre>
  
  <p>So, I am copying the exact same requests sent by Google Chrome, as you can see in the pics below:</p>
  
  <p><a href="https://i.stack.imgur.com/s6RoN.png" rel="nofollow noreferrer">Pic1: OPTIONS Request by Chrome</a></p>
  
  <p><a href="https://i.stack.imgur.com/vwZOg.png" rel="nofollow noreferrer">Pic2: POST request by Chrome</a></p>
  
  <p>The first time I was blocked, I wasn't using the COOKIEJAR option yet, so I figured that could be the issue. I enabled it, but still blocked. </p>
  
  <p>So I know my code works fine because it was working great before the block and even before enabling cookies at all. I haven't changed anything since then. But it just seems that they can identify that this request comes from curl and then they block it.</p>
  
  <p>So my question is: How are they blocking me if it's not through my IP? What differs my curl request from a normal browser request?</p>
  
  <p>My bet is that this has to do with the cookies, but I have no clue on how to handle this issue other than enabling the COOKIEJAR option on the curl request.</p>
  
  <p>Both of my cookie files are like this:</p>
  
  <p><a href="https://i.stack.imgur.com/ESUVS.png" rel="nofollow noreferrer">Pic3: Cookie Files</a></p>
  
  <p>Thanks.</p>