Web Parsing Course: Lesson 4 - Dealing with Anti-Scraping Mechanisms

Objective:

In this lesson, you will learn how to navigate websites that use anti-scraping measures, such as
CAPTCHAs, IP blocking, and bot detection. These mechanisms are designed to block automated
bots like scrapers, so it's essential to understand how to handle or bypass them in an ethical manner.

Lesson Outline:

1. Introduction to Anti-Scraping Mechanisms

o Why Do Websites Use Anti-Scraping Mechanisms?
 To protect against excessive load, competitive data scraping, and abuse.
 Examples include e-commerce sites, ticketing services, and social media
platforms.
o Common Anti-Scraping Tactics:
 CAPTCHAs (to distinguish humans from bots).
 IP blocking and rate limiting.
 JavaScript-based bot detection (e.g., monitoring browser behavior).
 Honey pots (invisible links meant to trap bots).
2. CAPTCHAs: Overview and Solutions
o What are CAPTCHAs?
 CAPTCHA stands for "Completely Automated Public Turing test to tell
Computers and Humans Apart."
 Different types: reCAPTCHA v2, reCAPTCHA v3, image-based
CAPTCHAs.
o Handling CAPTCHAs:
 Manual CAPTCHA Solving: Pausing the scraping process to solve
CAPTCHAs manually.
 Third-Party CAPTCHA Solving Services:
 Automated services that use human intervention to solve
CAPTCHAs.
 Example: 2Captcha, Anti-Captcha.
 Bypassing CAPTCHAs: Some sites only show CAPTCHAs when scraping
is detected. Using human-like interactions can reduce the chances of
encountering CAPTCHAs.
o Example using 2Captcha API:

python
Копіювати код
import requests

captcha_api_key = "your-2captcha-api-key"
captcha_response = requests.post("https://2captcha.com/in.php",
data={
'key': captcha_api_key,
'method': 'userrecaptcha',
'googlekey': 'site_key_from_target',
'pageurl': 'https://targetwebsite.com'
})

o Solving CAPTCHAs with Playwright:

 Playwright supports CAPTCHA interaction to some extent, but if CAPTCHA
solving is too complex, external services can help.
3. IP Blocking and Rate Limiting
o Why Do Websites Block IPs?
 To prevent scraping from a single source that sends too many requests in a
short time (rate-limiting).
o Rotating IPs with Proxy Services:
 Residential Proxies:
 Most robust but expensive; simulate real user traffic through
residential IPs.
 Examples: Luminati, Smartproxy.
 Datacenter Proxies:
 Faster and cheaper but more easily detected and blocked.
 Free vs Paid Proxies:
 Free proxies are unreliable and frequently blocked.
o Implementing Proxies in Python:

python
Копіювати код
from playwright.sync_api import sync_playwright

proxy = {"server": "http://proxyserver.com:port", "username":

"proxyuser", "password": "proxypass"}

with sync_playwright() as p:
browser = p.chromium.launch(proxy=proxy)
page = browser.new_page()
page.goto("https://example.com")
```

o Rotating Proxies Automatically:

 Use libraries like Scrapy-rotating-proxies to change IPs frequently.
 For Playwright:

python
Копіювати код
proxy_list = [
"http://proxy1.com:port",
"http://proxy2.com:port",
"http://proxy3.com:port"
]
for proxy in proxy_list:
browser = p.chromium.launch(proxy={'server': proxy})

4. User-Agent Spoofing
o What is a User-Agent?
 A string sent by browsers that identifies the browser and operating system to
the server.
o Why Spoof User-Agents?
 Websites can block specific bots by detecting their User-Agent. Spoofing
helps simulate a legitimate browser.
o How to Rotate User-Agents:
 A User-Agent can be changed by modifying the request headers.

python
Копіювати код
user_agents = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36",
 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2
Safari/605.1.15"
]
page.set_extra_http_headers({"User-Agent": user_agents[0]})

o Rotating User-Agents for Each Request:

 Rotate user-agents programmatically to mimic different users.

python
Копіювати код
from random import choice

user_agent = choice(user_agents)
page.set_extra_http_headers({"User-Agent": user_agent})

5. Other Headers and Browser Fingerprinting

o Headers That May Reveal Bots:
 Websites may check headers like Referer, Accept-Language, and Accept-
Encoding to detect abnormal requests.
o Simulating Real Browsers:
 Use Playwright to mimic browser behavior, including cookies, local storage,
and other details that real browsers use.

python
Копіювати код
page.set_extra_http_headers({
"Referer": "https://example.com",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br"
})

6. Delaying Requests and Randomized Intervals

o Why Add Delays?
 Rapid, successive requests from the same IP can flag your activity as bot-
like.
o How to Implement Random Delays:
 Use Python’s time.sleep() to add random intervals between requests.

python
Копіювати код
import time
from random import uniform

time.sleep(uniform(2, 5)) # Wait between 2 to 5 seconds

o Throttling Browser Automation:

 Playwright allows you to slow down the scraping process using its slow_mo
parameter:

python
Копіювати код
browser = p.chromium.launch(slow_mo=500) # Waits 500ms between
actions

7. Detecting and Avoiding Honey Pot Links

o What Are Honey Pot Links?
 Hidden or non-visible links placed on a webpage to trap bots. Humans won’t
interact with these, but bots might click them, revealing their presence.
 o Avoiding Honey Pot Links:
 Check for invisible elements and avoid clicking anything that is not visible to
human users.

python
Копіювати код
if not element.is_visible():
continue # Skip invisible elements

8. Monitoring Network Behavior

o Capturing and Analyzing AJAX Requests:
 Monitor and understand how the website communicates via network requests
(e.g., API calls, async requests).
o Handling Network Request Blocks:
 If a network request is blocked, capture the HTTP status codes and errors to
handle retries or alternatives.

python
Копіювати код
page.on("response", lambda response: print(response.status))

9. Simulating Human Interaction

o Randomized Mouse Movements and Scroll Behavior:
 Some websites use browser fingerprinting to detect bot-like mouse
movements and scrolling.

python
Копіювати код
page.mouse.move(x1, y1)
page.mouse.move(x2, y2)
page.mouse.wheel(deltaX=0, deltaY=100) # Simulate scrolling

o Interaction Patterns:
 Simulate common user actions like clicking buttons, typing slowly, or
hovering over elements:

python
Копіювати код
page.click("button.search")
page.hover("a.menu-item")
page.type("input.search", "Playwright", delay=100) # Type with a
100ms delay between keystrokes

10. Practical Task: Bypassing Simple Anti-Scraping Mechanisms

o Scenario: Scrape data from a website that implements basic anti-scraping techniques
(e.g., CAPTCHAs, IP blocking, or User-Agent blocking).
o Use proxies, rotating User-Agents, and randomized delays to scrape the data
successfully.
o Bonus: Try simulating human-like interactions (e.g., scrolls and clicks) and observe
whether the website changes its behavior toward your scraper.

Key Takeaways:

 Ethical scraping: Always ensure that scraping is done within the website's terms of service
and laws.
  Anti-scraping techniques can make scraping more challenging, but using proxies, human-
like behavior, and appropriate headers can help mitigate detection.
 Understanding CAPTCHAs, rate-limiting, and browser fingerprinting is crucial to avoid
blocking.

By the end of this lesson, you'll be equipped to deal with common anti-scraping mechanisms and
create scrapers that are harder to detect, making your scraping workflows more robust