🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
-
Updated
Sep 4, 2024 - TypeScript
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Run a high-fidelity browser-based crawler in a single Docker container
🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖
Spring Boot + Keycloak Backend / Angular Web App
🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.
A simple TypeScript framework for declaratively composing bots with Puppeteer
Web Scraper and Crawler for LLM Apps and AI Workflows with NoCode / LowCode. Plug and play with your own logic and customize it flexibly and scalably on BuildShip.
A web crawling library written in TypeScript.
HTML type document parser based on jQuery and JSDOM
A web crawler built using NestJS (based on BFS)
Awesome boilerplate for writing browser automations using Playwright, with debugging and tests ready to go.
🕸️ Web crawler with configurable task execution and visualization
Taipei Veterans General Hospital Medical Library collection crawler 「臺北榮民總醫院醫學圖書館」館藏資料爬蟲
Scan given website recursively and report 404 links
Add a description, image, and links to the web-crawler topic page so that developers can more easily learn about it.
To associate your repository with the web-crawler topic, visit your repo's landing page and select "manage topics."