Skip to content

Robots.txt Generator

Create a robots.txt draft with editable crawler rules, site presets, and optional sitemap URL.

Updated

What is the Robots.txt Generator?

Create a robots.txt file from common presets or custom rules.

Robots.txt gives compliant crawlers instructions about which URLs they should request. It does not protect private content or force crawlers to obey.

Start with a site-type preset, edit the rules, then review the generated text before publishing it at your site's root. AI crawler user-agent names are included as editable rules; behavior depends on each crawler's policy.

How it works

Use the Robots.txt Generator to draft a file from sample rules and customize it for your site.

  1. Choose a starting preset: Select a site type to load sample Disallow paths. These are not universal recommendations; check each path against your actual site.

  2. Add settings: Provide an optional sitemap URL and crawl-delay comment. Crawl delay is not supported by all crawlers.

  3. Add crawler rules: Toggle the listed crawler rules or edit them. Robots.txt is voluntary guidance; it cannot enforce access restrictions.

  4. Customize rules: Add or modify user-agent groups with Allow/Disallow paths. Confirm that directives are supported by the crawlers you target.

  5. Preview and validate: Review the generated text and basic syntax warnings. Validation does not check crawler-specific behavior.

  6. Export: Copy to clipboard or download the file. Place it in your site's root directory.

The tool creates text directives but does not check how a particular crawler interprets them or whether a rule blocks important resources. Test the final file and protect private content with server-side access controls.

Examples

WordPress Site

Start with the WordPress sample rules and review them against your site's actual paths.

Site Type: WordPress
Sitemap: https://example.com/sitemap.xml
Block AI Training: Yes
 
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
...
Sitemap: https://example.com/sitemap.xml

E-commerce Store

Draft crawler directives for selected storefront paths; this does not prevent public access.

Site Type: E-commerce
Custom paths: /cart/, /checkout/
Block AI: Yes
 
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /cart/
...

Custom AI-Focused Rules

Add directives for listed crawler names; crawlers may choose whether to follow them.

Block AI Training crawlers + custom User-agent rules
 
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Disallow:

Frequently asked questions

What is a robots.txt file?

A robots.txt file is usually placed at a site's root and gives compliant crawlers instructions about URLs they should not request. It does not prevent access, remove a URL from search results, or protect private content.

Is this robots.txt generator free?

Yes, the tool is free and does not require sign-up. Input is processed in your browser; avoid shared or untrusted devices for sensitive site details.

Does it support blocking AI crawlers?

The generator can add rules for listed user-agent names. A crawler may ignore robots.txt, user-agent names and policies can change, and this file does not technically block requests.

How do I use the generated robots.txt?

  1. Copy the generated content.
  2. Create a new file named exactly robots.txt (lowercase).
  3. Upload it to the root directory of your website (same level as your index.html or homepage).
  4. Test it using Google Search Console.

What are common mistakes this tool helps avoid?

Its basic checks can flag some formatting issues. It cannot confirm that rules match your site's needs, are interpreted identically by every crawler, or will not block useful resources; review the result before deployment.

Does it work well for WordPress sites?

The WordPress preset provides sample paths. Review them carefully: blocking resources such as stylesheets or scripts can affect how some crawlers render a page.

Can I add custom rules?

Use the custom rules editor to add directives. Check that each directive is supported by the crawlers and services you target.

Is robots.txt enough to stop all unwanted crawlers?

Robots.txt is a request, not enforcement. A URL disallowed from crawling may still appear in search results if discovered elsewhere. Use authentication or server-side access controls for private content.

How often should I update my robots.txt?

Review it after major site changes, new plugin installations, or when new AI crawlers emerge. Our tool makes quick updates easy.

Related tools

More utilities you might find handy.

SEO

Alt Text Checker

Review img alt attributes in pasted HTML for missing values and possible quality issues.
SEO

Canonical Tag Generator

Create a canonical link tag from a URL and review selected URL normalizations before use.
SEO

Header Tag Analyzer

Review H1–H6 heading structure in pasted HTML and create a readable outline.