InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now. Learn more →
Top 23 Python Scraping Projects
-
What I have seen it is hard to tell what "serious scrapers" use. They use many things. Some use this, some not. This is what I have learned reading webscraping on reddit. Nobody speaks things like that out loud.
There are many tools, see links below
Personally I think that running selenium can be a bottle neck, as it does not play nice, sometimes processes break, even system sometimes requires restart because of things blocked, can be memory hog, etc. etc. That is my experience.
To be able to scale I think you have to have your own implementation. Serious scrapers complain about people using selenium, or derivatives as noobs, who will come back asking why page X does not work in scraping mechanisms.
https://github.com/lexiforest/curl_cffi
https://github.com/encode/httpx
https://github.com/scrapy/scrapy
https://github.com/apify/crawlee
-
Stream
Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.
-
Jobs_Applier_AI_Agent_AIHawk
AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.
-
-
Project mention: Maigret – An open source OSINT tool for tracking people | news.ycombinator.com | 2025-06-12
-
-
undetected-chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
-
Scrapling
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
Project mention: Agentic Coding Slot Machines – Did We Just Summon a Genie Addiction? – Part 1 | news.ycombinator.com | 2025-07-03 -
InfluxDB
InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
-
crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Project mention: Launching Crawlee for Python v1.0 to simplify building web scrapers and crawlers | news.ycombinator.com | 2025-09-30 -
-
trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Project mention: Trafilatura: A tool and library to gather text and metadata on the Web | news.ycombinator.com | 2025-05-28 -
-
twikit
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
-
-
Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
-
-
-
Project mention: Use the Accept Header to Serve Markdown Instead of HTML to LLMs | news.ycombinator.com | 2025-09-28
HTML into Markdown isn't particularly hard. Two methods I use:
1. The Jina reader API - https://jina.ai/reader/ - add r.jina.ai to any URL to run it through their hosted conversion proxy, eg https://r.jina.ai/www.skeptrune.com/posts/use-the-accept-hea...
2. Applying Readability.js and Turndown via Playwright. Here's a shell script that does that using my https://shot-scraper.datasette.io tool: https://gist.github.com/simonw/82e9c5da3f288a8cf83fb53b39bb4...
-
cloudproxy
Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.
-
-
-
Yes, Parsel is open-source and freely available under the BSD license. You can find its source code on Parsel's Github page
-
Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers, user info, images...
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Scraping discussion
Python Scraping related posts
-
Selenium Navigation and Page Interaction
-
How I Block All 26M of Your Curl Requests
-
ScrapeGraphAI Release Week
-
The last six months in LLMs, illustrated by pelicans on bicycles
-
ScrapeGraphAI: You Only Scrape Once
-
Scrapy needs to have sane defaults that do no harm
-
How to scrape Bluesky with Python
-
A note from our sponsor - InfluxDB
www.influxdata.com | 16 Nov 2025
Index
What are some of the best open-source Scraping projects in Python? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | Scrapy | 58,941 |
| 2 | Jobs_Applier_AI_Agent_AIHawk | 29,062 |
| 3 | Scrapegraph-ai | 21,751 |
| 4 | maigret | 17,927 |
| 5 | requests-html | 13,853 |
| 6 | undetected-chromedriver | 12,001 |
| 7 | Scrapling | 8,110 |
| 8 | crawlee-python | 7,145 |
| 9 | autoscraper | 7,022 |
| 10 | trafilatura | 4,898 |
| 11 | fake-useragent | 4,011 |
| 12 | twikit | 3,719 |
| 13 | snoop | 3,496 |
| 14 | Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE | 3,253 |
| 15 | facebook-scraper | 2,769 |
| 16 | Grab | 2,428 |
| 17 | shot-scraper | 2,134 |
| 18 | cloudproxy | 1,621 |
| 19 | mlscraper | 1,359 |
| 20 | DataEngineeringProject | 1,326 |
| 21 | parsel | 1,287 |
| 22 | Scweet | 1,236 |
| 23 | parsera | 1,234 |
Sponsored
Stream - Scalable APIs for Chat, Feeds, Moderation, & Video.
Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.
getstream.io