From a1a8f29308de1d9a57258aa437907096214932d4 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Sun, 5 Jul 2026 11:21:10 +0200 Subject: [PATCH] Add crawlberg to Web Scraping --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index f514d35..34c453a 100644 --- a/README.md +++ b/README.md @@ -380,6 +380,7 @@ _Libraries to automate web scraping and extract web content._ - Frameworks - [browser-use](https://github.com/browser-use/browser-use) - Make websites accessible for AI agents with easy browser automation. - [crawl4ai](https://github.com/unclecode/crawl4ai) - An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents. + - [crawlberg](https://github.com/xberg-io/crawlberg) - A high-performance web crawling engine with a Rust core, headless-browser fallback, and built-in robots.txt and sitemap parsing. - [mechanicalsoup](https://github.com/MechanicalSoup/MechanicalSoup) - A Python library for automating interaction with websites. - [scrapy](https://github.com/scrapy/scrapy) - A fast high-level screen scraping and web crawling framework. - Content Extraction