robots.txt is a request, not a lock.
The file tells cooperative crawlers which paths they may request. It does not hide content, enforce permissions or guarantee de-indexing.
A useful minimum
A public site often needs a clear user-agent block, an allow rule, and a sitemap location. Keep rules precise. A broad Disallow: / can make an entire site invisible to crawlers.
What to keep separate
Use authentication or server controls for private material. Use a noindex directive where supported for indexation preferences, and remember that a crawler may need to fetch a page to see it. Put the sitemap at an absolute HTTPS URL.
Test before publishing
Check the file at https://your-domain.example/robots.txt, inspect redirects and confirm the sitemap links resolve. Review changes during a staging-to-production launch. Read our editorial policy for how this guide is maintained.