Plain definition
Definition of robots.txt
A plain-text file placed at the root of a host that tells automated web crawlers which URL paths they may or may not request under the Robots Exclusion Protocol.
User-agent: * | Allow: / | Disallow: /dashboard/ | Sitemap: https://example.com/sitemap.xmlWorked example
Standardized in September 2022 as IETF RFC 9309, the Robots Exclusion Protocol defines how crawlers fetch and parse /robots.txt on each host, protocol, and port. Say a made-up booking app called SlotQuill runs its marketing pages at https://slotquill.example/ and its authenticated customer workspace under /app/ and /api/. SlotQuill publishes a short robots.txt file at https://slotquill.example/robots.txt with User-agent: *, Allow: /, Disallow: /app/, Disallow: /api/, and a Sitemap line pointing to its XML sitemap.
When Googlebot visits the site, it fetches robots.txt first, typically caches it for up to 24 hours, and obeys the matching group rules while ignoring any content past Google's 500 KiB file-size limit. Because /app/ is disallowed, bots skip useless login redirects and focus on SlotQuill's landing pages and docs. Crucially, SlotQuill does not block its JavaScript or CSS bundles, because Googlebot needs those files to render pages properly.
Why it matters when you launch
One stray Disallow: / line carried over from a staging environment can silently hide your entire product from Google for weeks after launch. Checking your live production robots.txt right after DNS cutover takes ten seconds and prevents one of the most painful silent launch failures.
Just as important is knowing what robots.txt cannot do. It is a crawl-management convention, not an access-control system and not a deindexing tool. Malicious scrapers can ignore it, and if another site links to a disallowed URL, Google can still index the bare URL without reading the page content. Put private data behind real authentication, and use noindex when you want a crawlable page kept out of search results.
Common mistakes
- Deploying a staging rule like Disallow: / to production and accidentally blocking Googlebot from crawling the entire site on launch day.
- Blocking CSS, JavaScript, or image folders needed to render your pages, which prevents search engines from seeing your actual layout and content.
- Relying on Disallow rules in robots.txt to remove sensitive or duplicate pages from Google Search instead of using authentication or a noindex directive.
- Listing secret admin paths in robots.txt and assuming they are hidden, when robots.txt is publicly readable by anyone who requests it.
Tools and guides that use it
Free tool
Robots.txt GeneratorWrite a robots.txt that lets AI search find you, and choose which AI crawlers can train on your site.Free tool
AI Search Readiness CheckerAudit your startup's visibility for Perplexity, ChatGPT Search, and Claude with an actionable 10-point GEO checklist.Founder guide
The Next.js Technical SEO Launch Checklist (2026)What to check before and after launch day: crawlability, metadata, canonical URLs, structured data, sitemaps, internal links, Core Web Vitals and AI crawlers.Founder guide
How to Get Your App Found in ChatGPT, Perplexity and Google AI SearchWhat AI search actually reads, the crawler settings that quietly block you, and the pages that make assistants mention your product, without tricks or guarantees.Related glossary terms
Official sources
Pritam Ghosh runs Hauntboard and maintains this glossary.
Published · Updated
Building something real? Launch it in public.
A person reviews every launch on Hauntboard, and every view and visit it sends your site is published on your launch page. Free to submit.