What does Disallow: / do?

It asks every crawler to ignore the entire site.

Is robots.txt a security measure?

No, and treating it as one is a mistake.

Blocking keeps you out of the training data and out of AI answers, which is increasingly how people find things.

At the root of the domain, so example.com/robots.txt.

Write a robots.txt without the one line that hides your whole site.

Search engines

Paths to keep out, one per lineThis is not security. It asks politely, and the file is public, so listing a secret path here advertises it.

Block AI crawlers

Sitemap URLCrawl delay (optional)Seconds between requests. Google ignores this.

robots.txt

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout

Sitemap: https://example.com/sitemap.xml

Save this as robots.txt at the root of your domain. It only works there, not in a subfolder.

What does Disallow: / do?: It asks every crawler to ignore the entire site.
Is robots.txt a security measure?: No, and treating it as one is a mistake.
Does it stop a page appearing in Google?: Not reliably.
Should I block AI crawlers?: Blocking keeps you out of the training data and out of AI answers, which is increasingly how people find things.
Where does the file go?: At the root of the domain, so example.com/robots.txt.