6 docs tagged with "web-scraping"

ArchiveBox is a self-hosted open-source internet archiving solution that allows you to save and browse webpages offline. It enables users to store full snapshots of web pages, including HTML, media, PDFs, and more, preserving them for future reference. ArchiveBox supports various input formats such as bookmarks, RSS feeds, and browser history, making it easy to automate and manage archiving tasks. The web interface provides a structured way to browse archived content, perform full-text searches, and organize saved snapshots efficiently. Additionally, it offers integrations with other tools, supports multiple storage backends, and provides API access for advanced automation.

Browserless is a powerful self-hosted headless browser service designed to automate web interactions. It enables developers to run headless Chrome instances efficiently for web scraping, testing, PDF generation, and automation tasks. Browserless supports Puppeteer, Playwright, and Selenium, providing a scalable and secure environment for handling complex browser automation tasks. The application is optimized for performance, offering session management, request queuing, and robust security features. With Browserless, developers can offload browser-based automation to a dedicated service, reducing local resource consumption and improving workflow efficiency.

Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown or structured data. It crawls all accessible subpages and gives you clean data for each. No sitemap required. Featuring advanced scraping, crawling, and data extraction capabilities with LLM-ready formats, anti-bot mechanisms, dynamic content handling, and powerful customization options. Perfect for AI applications, data extraction, and web automation.

FlareSolverr is a proxy server designed to help bots and scripts bypass web protections like Cloudflare's anti-bot page. It acts as an intermediary that can execute JavaScript and handle cookies and sessions, making it appear as if requests are coming from a real web browser. This tool is particularly useful for web scraping and automated data collection tasks where access to JavaScript-heavy websites is required. FlareSolverr supports multiple platforms and can be integrated into existing scraping setups to solve CAPTCHAs and manage browser sessions efficiently.

Huginn is a system for building agents that perform automated tasks for you online. They can read the web, watch for events, and take actions on your behalf.

Knowledge management system that helps you collect, search, and retrieve information efficiently.