Esc
  • Jump to
  • The board
  • All launchesSort and filter by week or category
  • Open statsEvery number, in public
  • Maker storiesApps that stopped hiding
  • Submit a launch
  • AICategory
  • AnalyticsCategory
  • API & BackendCategory
  • Audio & VoiceCategory
  • AutomationCategory
  • CommunityCategory
↑↓ to move↵ to openPress / anywhere
Sign in

What is robots.txt?

A plain-text file placed at the root of a host that tells automated web crawlers which URL paths they may or may not request under the Robots Exclusion Protocol.

Updated · SEO & AI Search

Plain definition

Definition of robots.txt

A plain-text file placed at the root of a host that tells automated web crawlers which URL paths they may or may not request under the Robots Exclusion Protocol.

Formula / syntaxUser-agent: * | Allow: / | Disallow: /dashboard/ | Sitemap: https://example.com/sitemap.xml

Worked example

Standardized in September 2022 as IETF RFC 9309, the Robots Exclusion Protocol defines how crawlers fetch and parse /robots.txt on each host, protocol, and port. Say a made-up booking app called SlotQuill runs its marketing pages at https://slotquill.example/ and its authenticated customer workspace under /app/ and /api/. SlotQuill publishes a short robots.txt file at https://slotquill.example/robots.txt with User-agent: *, Allow: /, Disallow: /app/, Disallow: /api/, and a Sitemap line pointing to its XML sitemap.

When Googlebot visits the site, it fetches robots.txt first, typically caches it for up to 24 hours, and obeys the matching group rules while ignoring any content past Google's 500 KiB file-size limit. Because /app/ is disallowed, bots skip useless login redirects and focus on SlotQuill's landing pages and docs. Crucially, SlotQuill does not block its JavaScript or CSS bundles, because Googlebot needs those files to render pages properly.

Why it matters when you launch

One stray Disallow: / line carried over from a staging environment can silently hide your entire product from Google for weeks after launch. Checking your live production robots.txt right after DNS cutover takes ten seconds and prevents one of the most painful silent launch failures.

Just as important is knowing what robots.txt cannot do. It is a crawl-management convention, not an access-control system and not a deindexing tool. Malicious scrapers can ignore it, and if another site links to a disallowed URL, Google can still index the bare URL without reading the page content. Put private data behind real authentication, and use noindex when you want a crawlable page kept out of search results.

Common mistakes

  • Deploying a staging rule like Disallow: / to production and accidentally blocking Googlebot from crawling the entire site on launch day.
  • Blocking CSS, JavaScript, or image folders needed to render your pages, which prevents search engines from seeing your actual layout and content.
  • Relying on Disallow rules in robots.txt to remove sensitive or duplicate pages from Google Search instead of using authentication or a noindex directive.
  • Listing secret admin paths in robots.txt and assuming they are hidden, when robots.txt is publicly readable by anyone who requests it.

Tools and guides that use it

Official sources

Pritam Ghosh runs Hauntboard and maintains this glossary.

Published · Updated

Building something real? Launch it in public.

A person reviews every launch on Hauntboard, and every view and visit it sends your site is published on your launch page. Free to submit.

Launch free →

On the board

All →

Still hiding your app?

Put it on the board. Reviewed by a person, free to launch.

Launch free
Submit a launch