Robots.txt Generator

robots.txt is a text file at the root of a website (/robots.txt) that tells web crawlers which parts of the site they should not fetch. The tool above generates robots.txt content based on inputs; nothing leaves your machine.

What robots.txt does

Crawlers — Googlebot, Bingbot, GPTBot, etc. — fetch https://yoursite.com/robots.txt before crawling and parse it for rules. A simple example:

User-agent: *
Disallow: /admin/
Disallow: /search?
Disallow: /tmp/

User-agent: Googlebot
Allow: /tmp/whitelisted/

Sitemap: https://yoursite.com/sitemap.xml

• User-agent: identifies which crawler the rules apply to. * matches all.
• Disallow: lists path prefixes the crawler should not fetch.
• Allow: is an exception — overrides a parent Disallow:.
• Sitemap: points to a sitemap (one or more lines).

The format is loose. The standard (RFC 9309) was finalized in 2022; before that, behavior varied subtly across crawlers.

What robots.txt isn’t

It’s not a security mechanism. Three reasons:

1. Compliance is voluntary. Well-behaved crawlers respect it; malicious crawlers ignore it (or scan it as a list of “interesting” paths to attack).
2. It’s publicly readable. Anyone can fetch https://yoursite.com/robots.txt and see what you’re trying to hide.
3. Disallowed pages are still indexed if linked. Google may show a “no description available” listing for a disallowed URL because robots.txt blocks the crawl, but the URL itself is publicly known.

If you have content that must not be public, secure it with authentication. robots.txt is a hint, not a wall.

What it’s good for

• Saving crawler budget. Telling Googlebot not to crawl /search?... (which produces infinite combinations) keeps it focused on your real pages.
• Avoiding duplicate-content penalties. If you have two URLs serving the same content (with/without trailing slash, with/without query params), robots.txt plus canonical URLs keeps indexing clean.
• Excluding development or staging directories from public search.
• Pointing crawlers at sitemaps. The Sitemap: line tells crawlers where to find your URL list.
• Excluding AI crawlers. Adding User-agent: GPTBot / Disallow: / opts out of OpenAI’s training crawl. Other major AI crawlers have similar identifiers.

Common patterns

# Allow everything (default)
User-agent: *
Disallow:

# Disallow everything (uncommon)
User-agent: *
Disallow: /

# Disallow admin and API
User-agent: *
Disallow: /admin/
Disallow: /api/internal/

# Disallow query strings
User-agent: *
Disallow: /*?

# Disallow specific extensions
User-agent: *
Disallow: /*.pdf$

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

# Block common image scrapers but allow Google Images
User-agent: *
Disallow: /images/

User-agent: Googlebot-Image
Allow: /images/

The wildcard (*) and end-anchor ($) extensions are widely supported but not in the original 1994 spec; modern crawlers accept them.

Path matching

Disallow: /admin             ← matches /admin, /admin/, /admin-foo, /admin/foo/...
Disallow: /admin/            ← matches /admin/, /admin/foo/... (but NOT /admin or /admin-foo)
Disallow: /admin/$           ← matches only /admin/ exactly
Disallow: /*.pdf$            ← matches any path ending in .pdf
Disallow: /*?                ← matches any URL with a query string

The matching is prefix-based unless $ anchors the end. The trailing slash on directory paths is the most common gotcha — Disallow: /admin blocks more than just the admin directory.

Crawler-specific identifiers

Major crawlers use specific User-Agent strings. The robots.txt User-agent line is matched case-insensitively against a substring of the crawler’s User-Agent header.

Each line affects only matching User-Agents. To block a specific crawler, add a section with that token. To block all, use *.

Crawl-Delay (sometimes supported)

User-agent: Bingbot
Crawl-delay: 5

Tells the crawler to wait N seconds between requests. Not in the original spec; not honored by Google (you set crawl rate in Search Console instead). Some crawlers honor it; many don’t.

Sitemap directive

Sitemap: https://yoursite.com/sitemap.xml
Sitemap: https://yoursite.com/sitemap-images.xml

Multiple sitemaps allowed. The directive is independent of User-agent groups — it applies to all crawlers. See sitemap generator for the sitemap format.

Common pitfalls

• Disallowing / thinking it allows the homepage. Disallow: / blocks everything, including the homepage. Use Disallow: (empty) to allow all.
• Trailing slash on path patterns. Disallow: /admin blocks more than Disallow: /admin/. Be explicit.
• Case sensitivity in paths. robots.txt path matching is case-sensitive (URLs are technically case-sensitive). /Admin is different from /admin.
• Treating robots.txt as access control. It’s not. Use authentication.
• Forgetting that “Disallow” doesn’t always mean “don’t index.” A disallowed URL can still appear in search results if external sites link to it. To prevent indexing, use <meta name="robots" content="noindex"> on the page itself.
• Multiple files. robots.txt must live at the root. https://yoursite.com/some-path/robots.txt is ignored. Subdomains have their own robots.txt.
• Comments inside rules. # comments are valid but only on their own lines or at the end of a line. Avoid putting them in unexpected places.

For other crawler-relevant tools, see sitemap generator, meta tag generator.