How to add llms.txt to WordPress

January 14, 2024

8 min read

LLMS.txt Team

Introduction

WordPress powers over 40% of all websites on the internet, making it crucial for WordPress site owners to understand how to implement LLMS.txt files effectively. This comprehensive guide will walk you through multiple methods to add LLMS.txt support to your WordPress website, from simple manual uploads to automated solutions.

Method 1: Manual File Upload (Easiest)

Step 1: Generate Your LLMS.txt File

First, create your LLMS.txt file using our LLMS.txt Generator:

Enter your WordPress site URL
Add any custom disallow rules (e.g., /wp-admin, /wp-content/uploads)
Include your contact information
Generate and download the file

Step 2: Upload via FTP or File Manager

Upload the file to your WordPress root directory (the same folder where you find wp-config.php):

your-wordpress-site/
├── wp-admin/
├── wp-content/
├── wp-includes/
├── wp-config.php
├── index.php
└── llms.txt  ← Upload here

Step 3: Verify the File

Visit https://yoursite.com/llms.txt to confirm the file is accessible and displays correctly.

Method 2: Using Functions.php (Dynamic Generation)

For a more dynamic approach that automatically updates with your site content, add this code to your theme’sfunctions.php file:

<?php
// Add LLMS.txt support to WordPress
function serve_llms_txt() {
    if (isset($_SERVER['REQUEST_URI']) && $_SERVER['REQUEST_URI'] === '/llms.txt') {
        header('Content-Type: text/plain; charset=UTF-8');
        
        $site_title = get_bloginfo('name');
        $site_description = get_bloginfo('description');
        $site_url = home_url();
        $admin_email = get_option('admin_email');
        
        echo "# " . $site_title . "\n\n";
        echo "> " . $site_description . "\n\n";
        
        echo "## Contact\n";
        echo "- Email: " . $admin_email . "\n";
        echo "- Website: " . $site_url . "\n\n";
        
        echo "## Pages\n\n";
        
        // Get important pages
        $pages = get_pages(array(
            'meta_key' => '_wp_page_template',
            'hierarchical' => 1,
            'parent' => 0,
            'number' => 20
        ));
        
        foreach ($pages as $page) {
            echo "### " . $page->post_title . "\n";
            echo "URL: " . get_permalink($page->ID) . "\n";
            $excerpt = wp_strip_all_tags($page->post_excerpt);
            if (empty($excerpt) && !empty($page->post_content)) {
                $excerpt = wp_trim_words(wp_strip_all_tags($page->post_content), 20);
            }
            if (!empty($excerpt)) {
                echo $excerpt . "\n\n";
            } else {
                echo "\n";
            }
        }
        
        // Add recent blog posts
        $posts = get_posts(array(
            'numberposts' => 10,
            'post_status' => 'publish'
        ));
        
        if (!empty($posts)) {
            echo "## Recent Blog Posts\n\n";
            foreach ($posts as $post) {
                echo "### " . $post->post_title . "\n";
                echo "URL: " . get_permalink($post->ID) . "\n";
                $excerpt = wp_strip_all_tags($post->post_excerpt);
                if (empty($excerpt)) {
                    $excerpt = wp_trim_words(wp_strip_all_tags($post->post_content), 20);
                }
                echo $excerpt . "\n\n";
            }
        }
        
        echo "## Crawling Rules\n";
        echo "Disallow: /wp-admin\n";
        echo "Disallow: /wp-includes\n";
        echo "Disallow: /wp-content/uploads\n";
        echo "Disallow: /?s=\n";
        echo "Disallow: /search\n";
        
        exit;
    }
}
add_action('init', 'serve_llms_txt');

Important: Always backup your site before modifying functions.php. Consider using a child theme to prevent losing changes during theme updates.

Method 3: Creating a Custom Plugin

For a more maintainable solution, create a custom plugin. Create a new filewp-content/plugins/llms-txt-generator/llms-txt-generator.php:

<?php
/**
 * Plugin Name: LLMS.txt Generator
 * Description: Automatically generates LLMS.txt file for your WordPress site
 * Version: 1.0
 * Author: Your Name
 */

// Prevent direct access
if (!defined('ABSPATH')) {
    exit;
}

class LLMSTxtGenerator {
    
    public function __construct() {
        add_action('init', array($this, 'handle_llms_txt_request'));
        add_action('admin_menu', array($this, 'add_admin_menu'));
    }
    
    public function handle_llms_txt_request() {
        if (isset($_SERVER['REQUEST_URI']) && $_SERVER['REQUEST_URI'] === '/llms.txt') {
            $this->serve_llms_txt();
        }
    }
    
    public function serve_llms_txt() {
        header('Content-Type: text/plain; charset=UTF-8');
        
        $options = get_option('llms_txt_options', array());
        
        $site_title = get_bloginfo('name');
        $site_description = get_bloginfo('description');
        $site_url = home_url();
        $admin_email = get_option('admin_email');
        
        echo "# " . $site_title . "\n\n";
        echo "> " . $site_description . "\n\n";
        
        echo "## Contact\n";
        echo "- Email: " . $admin_email . "\n";
        echo "- Website: " . $site_url . "\n";
        
        if (!empty($options['additional_contact'])) {
            echo $options['additional_contact'] . "\n";
        }
        echo "\n";
        
        echo "## Pages\n\n";
        
        // Include selected pages
        $included_pages = isset($options['included_pages']) ? $options['included_pages'] : array();
        
        if (empty($included_pages)) {
            // Default to all published pages
            $pages = get_pages(array('post_status' => 'publish'));
            foreach ($pages as $page) {
                $included_pages[] = $page->ID;
            }
        }
        
        foreach ($included_pages as $page_id) {
            $page = get_post($page_id);
            if ($page && $page->post_status === 'publish') {
                echo "### " . $page->post_title . "\n";
                echo "URL: " . get_permalink($page->ID) . "\n";
                
                $excerpt = wp_strip_all_tags($page->post_excerpt);
                if (empty($excerpt) && !empty($page->post_content)) {
                    $excerpt = wp_trim_words(wp_strip_all_tags($page->post_content), 20);
                }
                if (!empty($excerpt)) {
                    echo $excerpt . "\n\n";
                } else {
                    echo "\n";
                }
            }
        }
        
        // Custom disallow rules
        $disallow_rules = isset($options['disallow_rules']) ? $options['disallow_rules'] : 
            "/wp-admin\n/wp-includes\n/wp-content/uploads\n/?s=\n/search";
            
        if (!empty($disallow_rules)) {
            echo "## Crawling Rules\n";
            $rules = explode("\n", $disallow_rules);
            foreach ($rules as $rule) {
                $rule = trim($rule);
                if (!empty($rule)) {
                    echo "Disallow: " . $rule . "\n";
                }
            }
        }
        
        exit;
    }
    
    public function add_admin_menu() {
        add_options_page(
            'LLMS.txt Settings',
            'LLMS.txt',
            'manage_options',
            'llms-txt-settings',
            array($this, 'admin_page')
        );
    }
    
    public function admin_page() {
        if (isset($_POST['submit'])) {
            $options = array(
                'additional_contact' => sanitize_textarea_field($_POST['additional_contact']),
                'disallow_rules' => sanitize_textarea_field($_POST['disallow_rules']),
                'included_pages' => isset($_POST['included_pages']) ? array_map('intval', $_POST['included_pages']) : array()
            );
            update_option('llms_txt_options', $options);
            echo '<div class="notice notice-success"><p>Settings saved!</p></div>';
        }
        
        $options = get_option('llms_txt_options', array());
        $pages = get_pages();
        
        ?>
        <div class="wrap">
            <h1>LLMS.txt Settings</h1>
            <form method="post">
                <table class="form-table">
                    <tr>
                        <th scope="row">Additional Contact Info</th>
                        <td>
                            <textarea name="additional_contact" rows="3" cols="50" class="large-text"><?php echo esc_textarea($options['additional_contact'] ?? ''); ?></textarea>
                            <p class="description">Additional contact information to include (e.g., phone, social media)</p>
                        </td>
                    </tr>
                    <tr>
                        <th scope="row">Disallow Rules</th>
                        <td>
                            <textarea name="disallow_rules" rows="5" cols="50" class="large-text"><?php echo esc_textarea($options['disallow_rules'] ?? "/wp-admin\n/wp-includes\n/wp-content/uploads\n/?s=\n/search"); ?></textarea>
                            <p class="description">One rule per line (without "Disallow: " prefix)</p>
                        </td>
                    </tr>
                    <tr>
                        <th scope="row">Include Pages</th>
                        <td>
                            <?php foreach ($pages as $page): ?>
                                <label>
                                    <input type="checkbox" name="included_pages[]" value="<?php echo $page->ID; ?>" 
                                           <?php checked(in_array($page->ID, $options['included_pages'] ?? array())); ?>>
                                    <?php echo esc_html($page->post_title); ?>
                                </label><br>
                            <?php endforeach; ?>
                            <p class="description">Select which pages to include in the LLMS.txt file</p>
                        </td>
                    </tr>
                </table>
                <?php submit_button(); ?>
            </form>
            
            <h2>Preview</h2>
            <p><a href="<?php echo home_url('/llms.txt'); ?>" target="_blank">View your LLMS.txt file</a></p>
        </div>
        <?php
    }
}

new LLMSTxtGenerator();

After creating this plugin, activate it from your WordPress admin panel and configure it underSettings → LLMS.txt.

WordPress-Specific Best Practices

Common Disallow Rules for WordPress

Here are recommended disallow rules for WordPress sites:

/wp-admin - WordPress admin area
/wp-includes - WordPress core files
/wp-content/uploads - Media files
/wp-content/plugins - Plugin files
/wp-content/themes - Theme files
/?s= - Search results pages
/search - Alternative search URLs
/author - Author archive pages (if not needed)
/tag - Tag pages (if not needed)
/category - Category pages (if not needed)

Performance Considerations

Caching: Consider caching the LLMS.txt output to reduce database queries
Limits: Limit the number of pages/posts included to keep the file manageable
Updates: Set up automatic regeneration when content changes

SEO Integration

Use your existing SEO plugin’s data for descriptions when available
Include canonical URLs to match your SEO setup
Consider excluding noindex pages from your LLMS.txt file

Troubleshooting Common Issues

File Not Accessible

Check file permissions (should be 644)
Ensure the file is in the correct directory
Clear any caching plugins
Check if rewrite rules are working

Content Not Updating

Clear WordPress cache
Clear server-side cache
Check if the dynamic generation code is working
Verify database queries are returning current data

Performance Issues

Implement caching for the LLMS.txt output
Reduce the number of included pages/posts
Optimize database queries
Consider using a static file for high-traffic sites

Advanced Features

Multilingual Support

If you’re using WPML or similar plugins, you can create language-specific LLMS.txt files:

/llms.txt - Default language
/en/llms.txt - English version
/es/llms.txt - Spanish version
/fr/llms.txt - French version

WooCommerce Integration

For WooCommerce sites, consider including:

Product categories
Popular products
Store policies
Contact and support information

Conclusion

Implementing LLMS.txt on WordPress sites is straightforward and offers multiple approaches depending on your technical comfort level and specific needs. Whether you choose the simple manual upload method or implement a dynamic solution, you’ll be making your WordPress site more AI-friendly and future-ready.

Start with the manual method to get familiar with LLMS.txt, then consider upgrading to a dynamic solution as your needs grow. Remember to test your implementation and keep your LLMS.txt file updated as your site evolves.