markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
Category
AI Writing
Quality
83/100
Primary source
GitHub
What is markdown-crawler?
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
Key features
Best fit
Why consider it
- markdown-crawler is categorized for ai writing workflows and tagged with Copywriting, SEO, Notes.
- The public repository has 455 stars, which gives buyers and builders an extra adoption signal.
- License metadata is available: MIT.
Source & verification
- Verified on Jun 30, 2026 from public source metadata.
- Primary reference: github.com.
- Repository freshness signal: last commit Jun 26, 2026.
Alternative tools
The API to search, scrape, and interact with the web at scale. 🔥
LlamaIndex is the leading document agent and OCR platform
A cross-platform Markdown AI note-taking software.
Related tools
📑 PageIndex: Document Index for Vectorless, Reasoning-based RAG
Qmedia
AI Writing
An open-source AI content search engine designed specifically for content creators. Supports extraction of text, images, and short videos. Allows full local deployment.
Kwipu
AI Writing
Ask questions across your Markdown notes using a fully local Graph RAG engine. Built for Obsidian vaults, works with any folder of Markdown files. Extracts.