Generate a clean, AI-friendly documentation file from any website. Extract content from sitemaps and create structured markdown for LLM consumption.
Comma-separated list of sitemap URLs. If not provided, common sitemap locations will be tried.
One path per line. URLs matching these paths will be excluded from crawling.
Contact information to be included in the generated documentation.
Retrieves the robots.txt file to respect crawling guidelines and find sitemaps.
Finds and parses XML sitemaps to discover all available pages.
Extracts titles, descriptions, and headings from up to 10 pages.
Creates a structured markdown file perfect for LLM consumption.
Everything you need to create comprehensive AI-friendly documentation from any website
Automatically finds and crawls sitemaps from common locations, or use your custom sitemap URLs.
Custom disallow rules to exclude admin pages, private sections, and unwanted content.
Generates clean, organized markdown perfect for LLM consumption and documentation.
Efficiently processes up to 100 pages with recursive sitemap crawling and SSL support.
Include contact information directly in the generated documentation for easy reference.
Preview the generated content in markdown or rendered format, then download as llms.txt.
Generate comprehensive documentation in just a few simple steps
Simply paste the website URL you want to document. Our system works with any publicly accessible website.
Set custom sitemap URLs, disallow rules, and contact information to customize the crawling process.
Our crawler discovers sitemaps, fetches page content, extracts titles and descriptions, and organizes everything automatically.
Preview the structured markdown output and download it as an llms.txt file ready for AI consumption.
Click on any example to see the llms.txt generator in action
React framework for production-grade applications
Typed superset of JavaScript that compiles to plain JavaScript