- Published on
Bypass Cloudflare
- Authors
- Name
- Shelton Ma
Browser Environment Validation
Example Website: Indeed US
- Target URL: Indeed Jobs
Using requests to fetch data from this URL directly will result in a 403 error with the following message:
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
To retrieve data successfully, consider simulating a browser environment. Using undetected chromedriver can bypass Cloudflare's checks effectively.
TLS and HTTP/2 Fingerprint Validation
Example Website: Marinetraffic
- Target URL: Marinetraffic Data
Observations
After Cloudflare was introduced, requests to this endpoint started returning 403 errors.
Even when providing complete cookies and headers, the requests failed. Interestingly, the same request worked in the Charles proxy tool but failed when sent via curl or a terminal. Packet capture comparisons revealed no significant differences.
Solution
Use the curl_cffi library and specify the impersonate parameter to simulate browser behavior, avoiding detection by Cloudflare.
from curl_cffi import requests
# Use the impersonate parameter
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome")
print(r.json())
# Pin a specific browser version
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome124")
# Provide custom JA3/Akamai fingerprints for non-browser impersonation
# See examples directory for details.
r = requests.get("https://tls.browserleaks.com/json", ja3=..., akamai=...)
# HTTP/SOCKS proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome", proxies=proxies)