From 0c992c59a6ad8ed825a6a2a02cea415dd57cd62f Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Sun, 5 Jul 2026 18:07:08 +0200 Subject: [PATCH] Add crawlberg to Web Scraping (#3238) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 6a4d13d..356379b 100644 --- a/README.md +++ b/README.md @@ -381,6 +381,7 @@ _Libraries to automate web scraping and extract web content._ - Frameworks - [browser-use](https://github.com/browser-use/browser-use) - Make websites accessible for AI agents with easy browser automation. - [crawl4ai](https://github.com/unclecode/crawl4ai) - An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents. + - [crawlberg](https://github.com/xberg-io/crawlberg) - A high-performance web crawling engine with a Rust core, headless-browser fallback, and built-in robots.txt and sitemap parsing. - [mechanicalsoup](https://github.com/MechanicalSoup/MechanicalSoup) - A Python library for automating interaction with websites. - [scrapy](https://github.com/scrapy/scrapy) - A fast high-level screen scraping and web crawling framework. - Content Extraction