Glossary

What is robots.txt?

robots.txt is a plain text file at the root of a website (yoursite.com/robots.txt) that tells crawlers, such as Googlebot or AI bots like GPTBot and ClaudeBot, which parts of the site they are allowed to fetch.

Also called robots exclusion protocol

How it works

The file is made of groups. Each group names a crawler with a User-agent line, then lists Allow and Disallow rules for URL paths. A group for “*” applies to every crawler that has no group of its own. The rules were used informally from 1994 and became an official internet standard, RFC 9309, in 2022.

robots.txt is a request, not a lock. Well-behaved crawlers from search engines and major AI companies follow it; anything that must stay private needs a login or a noindex tag, not a robots.txt rule.

Crawling is not indexing

Blocking a URL in robots.txt stops crawlers from reading it, but Google can still list the bare URL if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex tag instead.

Why it matters for B2B websites

AI companies use named crawlers, and robots.txt is where you decide whether they can read your site. Examples include GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, and Google-Extended, a token that controls whether Google may use your content for its Gemini models.

Many sites block these bots by accident, through a security plugin, a hosting default or a copied template. A B2B company that wants to be recommended by AI assistants has to let them read its pages first.

Example

  • User-agent: *
  • Allow: /
  • Disallow: /preview/
  • Sitemap: https://example.com/sitemap.xml

How we handle it

We write robots.txt on purpose rather than leaving the default: search engines and AI crawlers named and allowed, drafts and private paths closed, and the sitemap listed. pentaclay.com’s own robots.txt allows every major search and AI crawler. Checking this file is one of the first steps in our AI search optimization work.

Last updated 15 November 2026

Related terms

Terms that go with robots.txt

All glossary terms

Questions about robots.txt

Can’t find your answer?

Book a call and let’s talk.

Book a call

Or email us at contact@pentaclay.com

Should I block AI crawlers in robots.txt?

It is a business choice. Blocking them keeps your content out of some AI training and answers, but it also means assistants are less likely to describe or recommend you. Most B2B companies that want to be found choose to allow them.

Does robots.txt stop a page from appearing in Google?

Not reliably. It stops crawling, but a blocked URL can still be indexed from links. Use a noindex meta tag on a crawlable page to keep it out of results.

Where does robots.txt have to be?

At the root of the host it applies to, such as https://example.com/robots.txt. A subdomain like blog.example.com needs its own file.

Can a robots.txt mistake hurt my rankings?

Yes. A single “Disallow: /” under “User-agent: *” blocks every compliant crawler from the whole site. This often happens when a staging site’s settings are copied to production.

Your business grew up. Your website should too

If customers need a demo just to understand what you sell, your website isn’t doing its job.

  • We sign an NDA on request.
  • You own the site, the code and every account once it’s paid.
  • Your first call is with Erfan, the founder.

Kazi Mohammed ErfanFounder

What do you need?
Budget

Don’t want to fill in the form? Book a call directly

Not sure? Ask AI about Pentaclay

Pentaclay, in short

Pentaclay is a B2B website design agency for AI companies that sell to enterprises. We write, design and build marketing websites in Framer, Webflow and Next.js, and stay to help after launch. Pentaclay LLC is registered in New Mexico, USA; our team works from Dhaka, Bangladesh, with calls until 12:30pm US Eastern. Founded in 2023 by Kazi Mohammed Erfan, who brings 12 years of design experience.