How AI Search Finds Products
When someone asks ChatGPT, Perplexity, Claude or Google "what's a good tool for X?", the answer increasingly comes from a live web search, not just from what the model memorised in training. The assistant searches, reads a handful of pages, and writes an answer that cites some of them.
That's good news for a new product. You don't need to wait for the next model to be trained on you. You need to be findable (crawlers can reach your pages), readable (the pages say plainly what you are) and corroborated (other sites say it too).
Nobody can promise that an assistant will recommend your product, and anyone selling a guarantee is selling something else. What you can do is remove every reason for it not to.
Don't Block the Crawlers
The most common reason a product never appears in AI answers is that its own site turns the crawlers away. Each assistant uses named crawlers, and your robots.txt decides whether they can read your pages:
| Assistant | Crawlers to allow for search and citations |
|---|---|
| ChatGPT search | OAI-SearchBot, ChatGPT-User |
| Claude | Claude-SearchBot, Claude-User |
| Perplexity | PerplexityBot |
| Google AI Overviews | Googlebot (the same crawler as normal search) |
Some crawlers are used to collect training data rather than to answer searches, and many sites block those on purpose. That's a legitimate choice. Just make sure you aren't blocking the search crawlers by accident. Check three places:
- Your robots.txt. Open
yourapp.com/robots.txt. A blanketDisallow: /underUser-agent: *blocks everyone. - Your host or CDN's bot settings. Several hosting and security providers offer one-click options to block AI bots. If you switched one on, it may be blocking search crawlers along with training ones.
- Your pages without JavaScript. Some crawlers don't run JavaScript. If your homepage is an empty shell until scripts load, they see nothing. Server-rendered or static pages are read reliably.
Pages Assistants Can Actually Use
Assistants answer questions, so they favour pages that answer questions plainly. Most landing pages are written to persuade, not to inform, and say very little a machine (or a person) can quote. Make sure your site has:
- A homepage that says what it is in the first sentence. "Acme is a bookkeeping app for solo founders" can be quoted. "Unlock your financial potential" can't.
- A pricing page in plain text. Prices shown only as images, or hidden behind "contact us", can't be used in an answer about cost.
- Use-case pages. One page per job your product does ("invoicing for freelance designers") matches the exact questions people ask.
- Honest comparison pages. "Acme vs Bigger Competitor" and "Bigger Competitor alternatives" are some of the most asked questions in any category. Be fair to the competitor, or the page won't be trusted.
- An FAQ that answers real questions. Does it work offline? Is there a free plan? Can I export my data? Short, direct answers get quoted.
- Docs or a changelog, if you have them. They show the product is real and maintained.
Write facts, not adjectives. Assistants tend to skip claims they can't verify, and readers do too.
Structured Data and llms.txt
Structured data is a small block of JSON in your page that describes it in a standard vocabulary (schema.org). For a software product, the SoftwareApplication type with a name, description, category and price helps Google understand the page, and it costs you ten minutes. Add Organization on your homepage and FAQPage where you have a real FAQ.
llms.txt is a proposed convention: a plain-text file at yourapp.com/llms.txt that summarises your site and links your most important pages, written for AI tools. It's an emerging idea rather than an official standard, and it's unclear how much any assistant relies on it today. It's cheap to add, so it's worth doing, but it's not a substitute for good pages. Hauntboard publishes one at /llms.txt if you want an example.
Mentions on Other Sites
An assistant deciding what to recommend looks for more than your own site's opinion of itself. It reads lists, reviews, directories and discussions, and a product that several independent sources describe the same way is easier to recommend with confidence.
- Reviewed launch boards and directories. A listing on a site that checks submissions gives you a permanent, crawlable page describing your product in neutral words. Every launch on Hauntboard gets one, reviewed by a person, open to the AI search crawlers above, and marked up with
SoftwareApplicationdata. - "Best X for Y" lists. Find the lists that already rank for your category and ask the authors, politely, to consider your product. Offer something useful: a free account to test, a fact they got wrong about a competitor.
- Genuine discussions. Helpful answers in communities where your users ask questions, with your product mentioned only where it truly fits.
- Your own content. A guide that solves your users' problem gets linked and quoted, and brings the product along with it.
Consistency matters. If your tagline is different on every site, you look like three different products. Use the same one-line description everywhere.
How to Check Where You Stand
Once a month, ask the assistants the questions your customers would ask, in their words:
- "What's a good [category] tool for [your user]?"
- "[Biggest competitor] alternatives for [your niche]"
- "What is [your product]?"
Note whether you're mentioned, which pages are cited, and what's said about you. If the answer to "what is [your product]?" is wrong or vague, the pages it cites are telling you what to fix. Answers vary from one run to the next, so look for patterns over a few tries rather than reacting to one.
What Not to Do
- Don't publish hidden text or keyword-stuffed pages for bots. Search engines treat it as spam, and it can cost you your normal rankings too.
- Don't fake reviews or mentions. Beyond being dishonest, a pattern of identical praise on low-quality sites is easy to spot.
- Don't mass-produce thin pages. Fifty near-identical "best tool for [city]" pages are more likely to drag your site down than lift it.
- Don't ignore normal SEO. Google's AI Overviews build on the same index as regular search. A page that ranks well and answers clearly is the best foundation for both.
Start with the basics in our technical SEO launch checklist, then get your first independent mentions by launching in the right places.