---
title: "AEO Best Practices: 7 Technical Checks for AI Search and GEO"
description: "AEO and GEO best practices: check what GPTBot and ClaudeBot see without JavaScript, keep facts consistent across schema and llms.txt, and measure AI visibility."
url: "https://vindler.solutions/blog/aeo-best-practices-ai-crawler-checks"
locale: "en"
updatedAt: "2026-11-17"
---

[Back to Blog](https://vindler.solutions/blog)

AEO Best Practices: Seven Checks on How AI Assistants Read Your Site

[AEO](https://vindler.solutions/blog?tag=AEO)[GEO](https://vindler.solutions/blog?tag=GEO)[Answer Engine Optimization](https://vindler.solutions/blog?tag=Answer%20Engine%20Optimization)[Generative Engine Optimization](https://vindler.solutions/blog?tag=Generative%20Engine%20Optimization)[AI Search](https://vindler.solutions/blog?tag=AI%20Search)[AI Visibility](https://vindler.solutions/blog?tag=AI%20Visibility)[Structured Data](https://vindler.solutions/blog?tag=Structured%20Data)[llms.txt](https://vindler.solutions/blog?tag=llms.txt)[Technical SEO](https://vindler.solutions/blog?tag=Technical%20SEO)[AI Crawlers](https://vindler.solutions/blog?tag=AI%20Crawlers)[ChatGPT](https://vindler.solutions/blog?tag=ChatGPT)[Claude](https://vindler.solutions/blog?tag=Claude)[Perplexity](https://vindler.solutions/blog?tag=Perplexity)

# AEO Best Practices: Seven Checks on How AI Assistants Read Your Site

Answer engine optimization (AEO), also called generative engine optimization (GEO), is mostly a question of what an AI crawler can read on your site and whether the facts it finds agree with each other. Seven technical checks you can run with curl, the failures they catch, and how to measure whether ChatGPT, Claude, Perplexity and Gemini name you.

November 17, 2026

1 minute

Share:

A buyer asks ChatGPT which firms build AI products, or asks Claude how much a tool costs. The assistant answers from what it learned in training, or it runs a search, fetches a handful of pages and writes an answer from what it read. Answer engine optimization (AEO) is the work of making that answer name you and get the facts right. The same work also goes by generative engine optimization (GEO), LLM SEO and AI search optimization.

Much of the AEO advice in circulation is SEO advice under a new name. Google's AI Overviews and AI Mode draw on Google's own index, so ordinary SEO already covers most of that surface. ChatGPT, Claude and Perplexity run their own crawlers and read pages differently, and the AEO best practices that are specific to them are technical. Most of them can be checked from a terminal in an afternoon.

In August we ran the test on ourselves. We asked Claude, with web search on, whether it knew Vindler. It had no prior knowledge, searched, and described us as based only in São Paulo, citing a RocketReach profile and an old domain reference. It also found an unrelated cybersecurity company with the same name on Crunchbase and warned the buyer not to mix us up. It corrected the location only after fetching our About page, which lists the New York office as well. Every one of those errors came from a source we control or could correct, and the checks below are the ones we have run on our own site since then.

## 1\. Check what AI crawlers see without JavaScript

The major AI crawlers do not run JavaScript. Vercel and MERJ analysed crawler traffic across Vercel's network in late 2024 and found that none of them rendered JavaScript, including OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Meta-ExternalAgent and ByteDance's Bytespider. GPTBot and ClaudeBot download JavaScript files and never execute them. Googlebot does render JavaScript, which is why a page can look perfect in Google's tools and still be close to empty for ChatGPT.

The check is to fetch the raw HTML and look for what matters:

```
# Is the structured data in the HTML the server sends?
curl -s -A "GPTBot" https://example.com/pricing | grep -c 'application/ld+json'
 
# Is the price in the text a crawler sees?
curl -s https://example.com/pricing | sed 's/<[^>]*>//g' | grep -i -E '\$[0-9]+'
```

The usual failure is JSON-LD added by a tag manager or a client component after the page loads, or an FAQ section that renders from an API call. The Schema.org validator and Google's Rich Results Test both run JavaScript, so they report the markup as valid while the AI crawlers get none of it. The fix is to render the content and the structured data on the server, or prerender the page at build time. Our own site is a React application, and every page is prerendered to static HTML for this reason.

## 2\. Keep facts consistent across pages, schema and llms.txt

An assistant writing an answer about your product reads several sources at once: the page copy, the JSON-LD on that page, your `llms.txt`, any page you wrote for AI systems, and third-party profiles. When they disagree, it picks one of them, and you do not get to choose which.

Pricing is where we see this most often. The pricing page shows a monthly and a yearly price, the schema lists only the lower yearly figure without saying it is billed yearly, and a summary page elsewhere mixes the two. An assistant asked "how much does it cost" can then quote any of three numbers, and every one of them looks sourced.

Make a list of the facts a buyer would ask about (prices with their billing period, plan names, product names, office locations, founding date, team size) and grep each one across every surface that states it. Better still, keep each fact in one data file and generate the page, the schema and `llms.txt` from it, so a price change cannot reach one copy and miss another. Wherever a number needs a qualifier, such as "billed yearly" or "per seat", put the qualifier next to the number in every copy.

## 3\. Keep structured data (JSON-LD) in step with the page

Structured data is written once, at launch, and then left alone while the page around it is redesigned. A year later the JSON-LD still describes features that were renamed or retired, and an assistant that trusts the schema describes the old product.

Add the JSON-LD to the review checklist for any copy change, and check two details that validators accept but crawlers handle badly. URLs in `url`, `@id` and `sameAs` should be absolute (`https://example.com/ai-info`, never `/ai-info`). The Organization entity should carry the same `@id` on every page, with `sameAs` pointing at your LinkedIn page, Crunchbase profile and Wikidata item, so that every page describes one company. We added a Person entity for the founder on the About page and linked blog authors to it by `@id` for the same reason.

## 4\. Write robots.txt for the AI crawlers you mean

Each AI vendor runs several crawlers with different jobs. OpenAI separates GPTBot (training), OAI-SearchBot (the ChatGPT search index) and ChatGPT-User (pages fetched while answering a user). Anthropic separates ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requests). Perplexity has PerplexityBot and Perplexity-User. Blocking a training crawler does not require blocking the search crawler that decides whether you appear in answers, and many sites that meant to opt out of training also removed themselves from AI search.

Two less obvious problems come up often. Under RFC 9309, a crawler follows only the most specific group that matches its name, and that group is not merged with the `User-agent: *` rules. A file with a short `User-agent: GPTBot` / `Allow: /` group and all the `Disallow` lines under `*` lets GPTBot crawl everything those lines were meant to protect. Repeat the disallow rules in every named group.

The second problem sits outside robots.txt. Since July 2025 Cloudflare blocks known AI crawlers by default on new domains, and other CDNs and WAFs have bot rules of their own, so a robots.txt that allows ClaudeBot does nothing if the edge answers it with a 403. A quick check with the crawler's user agent catches the obvious cases:

```
for ua in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot; do
  printf '%-18s %s\n' "$ua" "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/)"
done
```

A spoofed user agent is not always treated the same way as the verified bot, so confirm in the CDN's bot analytics or the server logs that the real crawlers are getting 200 responses.

## 5\. Add llms.txt, a facts page and markdown versions

`llms.txt`, proposed by Jeremy Howard of Answer.AI in September 2024, is a markdown file at the root of the site that summarises what the site is and links to the pages that matter. No major assistant has confirmed that it reads the file when answering, so treat it as cheap insurance and not as a ranking factor. Since it is one more place facts are stated, it falls under check 2: we found our own `llms.txt` repeating an old positioning line that the rest of the site had already dropped.

Two related steps cost little and remove guesswork for any agent that does fetch the site. One is a single facts page (ours is [/ai-info](https://vindler.solutions/ai-info)) with the company facts in plain sentences, linked from `llms.txt` and from the footer so crawlers find it. The other is a markdown version of each page: every page on vindler.solutions is also served as markdown at its path plus `.md`, and to any request that sends `Accept: text/markdown`. We also send a `Content-Signal` line in robots.txt and as a response header to state what the content may be used for (search, AI answers, training).

## 6\. Fix your entity on third-party sources

Our location error came from a data broker profile, and the name collision came from Crunchbase, so the fixes were off our site. We created a Crunchbase profile and a Wikidata item, linked both from the Organization schema, corrected the profiles we could reach, and added the offices to the About page in plain text.

Reviews on third-party platforms carry weight in answers. In the same test, Claude told the buyer to check whether our Clutch profile had real reviews, and when it recommended firms it said it favoured ones with public case studies that named their clients. It dismissed "top 10 companies" listicles as paid placements. A named client review on a platform the assistant already trusts is worth more than another page on your own site saying the same thing.

## 7\. Measure AI visibility in ChatGPT, Claude, Perplexity and Gemini

Pick a small fixed set of prompts: one where a buyer asks for recommendations in your category without naming you, and one that asks about you directly. Run them each quarter in a fresh session with web search on, in ChatGPT, Claude, Perplexity and Gemini. For each run, record whether you were named, how you were described, which sources the assistant cited, and the reasons it gave for the firms it did name. Asking the assistant why it picked those firms is the most useful prompt in the set, because its answer tells you which evidence it weighs.

Alongside the prompts, track referral traffic from chatgpt.com, claude.ai, perplexity.ai and gemini.google.com as a channel of its own in your analytics, and count crawler hits by user agent in the server logs. The prompts show what assistants say about you, the referrals show whether anyone acts on it, and the logs show whether the crawlers can reach you at all.

## Where to start

Start with checks 1 and 2, because they make answers wrong, and a wrong price or location quoted by an assistant is worse than not being mentioned. Checks 4 and 6 come next, since they decide whether you appear at all. Run the measurement in check 7 before changing anything so you have a baseline to compare against.

The next step after being read correctly is being used. Assistants and browsers increasingly act on websites as well as reading them, and WebMCP lets a site expose its actions to an agent as tools with schemas. We describe that work on the [Agent Readiness Sprint](https://vindler.solutions/agent-ready) page. If you want help with how AI systems read or use your product, [book a call](https://vindler.solutions/book).

## Frequently Asked Questions

**What is answer engine optimization (AEO)?** AEO is the work of making AI assistants such as ChatGPT, Claude, Perplexity and Gemini describe your company or product correctly and name it in their answers. In practice it covers what their crawlers can read on your site, whether the facts they find agree, and which third-party sources they trust.

**What is the difference between AEO, GEO and SEO?** AEO and GEO (generative engine optimization) are two names for the same goal. SEO aims at a ranked list of links, and AEO aims at a written answer that may cite a few sources or none. Most SEO groundwork still applies. The differences are that AI crawlers do not run JavaScript, that each vendor runs separate crawlers for training, search and user requests, and that an assistant combines facts from several pages into one answer.

**Does llms.txt help with AEO?** It is cheap to add and no major assistant has confirmed reading it at answer time. Treat it as one more copy of your facts, which means it has to agree with the rest of the site.

**Should I block GPTBot and ClaudeBot?** Blocking the training crawlers (GPTBot, ClaudeBot) keeps your future pages out of model training. It does not remove you from AI search as long as the search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) are allowed, in robots.txt and at the CDN.

**How do I check whether ChatGPT can read my website?** Fetch the page with curl and a crawler user agent, then look for your key text and your JSON-LD in the raw HTML. If they only appear after JavaScript runs, ChatGPT's crawlers do not see them. Then check that your CDN returns 200 to the real crawlers in its bot analytics or your server logs.

**How do you measure AEO results?** Run a fixed set of prompts each quarter in ChatGPT, Claude, Perplexity and Gemini with web search on, and record whether you are named, how you are described and which sources are cited. Track referral traffic from chatgpt.com, claude.ai, perplexity.ai and gemini.google.com as its own channel.

## References

1.  Vercel (2024). [The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler). Crawler traffic analysis with MERJ.
2.  OpenAI. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots).
3.  Anthropic. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
4.  Koster, M., Illyes, G., Zeller, H., Sassman, L. (2022). [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309).
5.  Howard, J. (2024). [The /llms.txt file](https://llmstxt.org/).
6.  Cloudflare (2025). [Content Independence Day: no AI crawl without compensation](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/).
7.  [Content Signals](https://contentsignals.org/).

Share:

Carlos Dutra, founder of Vindler Solutions

### Carlos Dutra

Founder of Vindler Solutions, where I help organizations put AI into production and design the operating model around it. I write about AI adoption, agent architectures, and what actually changes inside a company once these systems ship. Completed Leading the AI-Driven Organization at MIT Sloan Executive Education.

[Get in Touch](https://vindler.solutions/contact)

## Related Posts

[What an LLM Request Actually Costs in Production: OpenAI, Claude and Gemini Compared per Task

LLM CostAI Cost OptimizationOpenAIClaudeGeminiPrompt CachingBatch APIModel Selection

### What an LLM Request Actually Costs in Production: OpenAI, Claude and Gemini Compared per Task

Price per million tokens tells you very little about your bill. We priced three real task shapes, a RAG support answer, an eight-call agent run and a document extraction, on ten current models from OpenAI, Anthropic and Google, with and without prompt caching and batch. The model tier you pick moves the cost a hundredfold. Caching and batch move it by a quarter to a half.

November 6, 2026

1 minute](https://vindler.solutions/blog/llm-api-cost-per-task-2026)[One Setting That Makes Claude Code Dramatically Faster at Navigating Code

ClaudeClaude CodeAIDeveloper ToolsProductivity

### One Setting That Makes Claude Code Dramatically Faster at Navigating Code

A single environment flag unlocks LSP support in Claude Code, replacing slow text searches with the same semantic code intelligence your IDE uses. Two minutes of setup, permanent improvement on any non-trivial codebase.

March 12, 2026

1 minute](https://vindler.solutions/blog/claude-code-lsp)[How to Make Claude Code /Insights Actually Work

AIClaudeClaude CodeEngineeringDeveloper ToolsProductivity

### How to Make Claude Code /Insights Actually Work

Everyone writes about getting roasted by /insights. Nobody writes about making it useful. Seven practical techniques for getting actionable results from Claude Code's undocumented self-assessment tool.

February 17, 2026

1 minute](https://vindler.solutions/blog/claude-code-insights-tailoring-guide)

### Subscribe to our newsletter

Get notified when we publish new posts on AI development, AWS, and software engineering.

## Structured data

```json
{"@context":"https://schema.org","@type":"Organization","@id":"https://vindler.solutions/#organization","name":"Vindler Solutions","alternateName":"Vindler","legalName":"Vindler Desenvolvimento de Sistemas de IA Ltda - EPP","url":"https://vindler.solutions","logo":"https://vindler.solutions/logo.png","description":"Vindler is a global AI engineering consultancy specializing in custom AI solutions, LLM integration, and enterprise-grade AI systems.","foundingDate":"2025-06-02","foundingLocation":{"@type":"Place","name":"São Paulo, Brazil"},"taxID":"61.102.920/0001-05","email":"contact@vindler.solutions","telephone":"+1-646-846-5339","numberOfEmployees":{"@type":"QuantitativeValue","minValue":1,"maxValue":10},"areaServed":[{"@type":"Country","name":"United States"},{"@type":"Place","name":"Europe"},{"@type":"Country","name":"Brazil"}],"knowsAbout":["Multi-agent systems","Retrieval-augmented generation","Voice AI agents","LangChain","LangGraph","LiveKit","AWS Bedrock","Model Context Protocol","LLM observability and evaluation"],"founder":{"@type":"Person","@id":"https://vindler.solutions/about#carlos-dutra","name":"Carlos Dutra","jobTitle":"Founder and CEO","url":"https://vindler.solutions/about","sameAs":["https://www.linkedin.com/in/carlosdutra","https://github.com/cdutr"]},"contactPoint":{"@type":"ContactPoint","contactType":"Customer Support","email":"contact@vindler.solutions","telephone":"+1-646-846-5339","url":"https://vindler.solutions/contact","availableLanguage":["English","Portuguese","Spanish"]},"address":[{"@type":"PostalAddress","addressLocality":"São Paulo","addressRegion":"SP","addressCountry":"BR","streetAddress":"Av. Paulista, 1636"},{"@type":"PostalAddress","addressLocality":"New York","addressRegion":"NY","addressCountry":"US","streetAddress":"524 Broadway","postalCode":"10012"}],"sameAs":["https://www.linkedin.com/company/vindler-tech/","https://github.com/vindler-solutions","https://clutch.co/profile/vindler-solutions","https://www.wikidata.org/wiki/Q141018871","https://www.crunchbase.com/organization/vindler-solutions","https://www.g2.com/products/vindler-solutions/reviews","https://www.designrush.com/agency/profile/vindler-solutions"],"memberOf":{"@type":"Organization","name":"Brazilian-American Chamber of Commerce, Inc.","url":"https://brazilcham.com/","sameAs":"https://www.linkedin.com/company/brazilian-american-chamber-of-commerce-inc./"},"hasOfferCatalog":{"@type":"OfferCatalog","name":"AI Engineering Services","itemListElement":[{"@type":"Offer","itemOffered":{"@type":"Service","name":"AI Engineering","description":"Custom AI systems development from prototyping to deployment."}},{"@type":"Offer","itemOffered":{"@type":"Service","name":"Custom ChatBots & Agents","description":"Intelligent, multilingual, and domain-specific conversational AI."}},{"@type":"Offer","itemOffered":{"@type":"Service","name":"AI Architecture Consultancy","description":"Designing scalable, secure, and ethical AI infrastructure for enterprises."}},{"@type":"Offer","itemOffered":{"@type":"Service","name":"Training & Team Building","description":"Upskill your team or recruit top-tier AI talent."}}]}}
```

```json
{"@context":"https://schema.org","@type":"BlogPosting","headline":"AEO Best Practices: Seven Checks on How AI Assistants Read Your Site","description":"Answer engine optimization (AEO), also called generative engine optimization (GEO), is mostly a question of what an AI crawler can read on your site and whether the facts it finds agree with each other. Seven technical checks you can run with curl, the failures they catch, and how to measure whether ChatGPT, Claude, Perplexity and Gemini name you.","image":"https://vindler.solutions/images/blog/aeo-crawler-checks-nano-v3.jpg","datePublished":"2026-11-17","dateModified":"2026-11-17","author":{"@type":"Person","@id":"https://vindler.solutions/about#carlos-dutra","name":"Carlos Dutra","jobTitle":"Founder and CEO, Vindler","url":"https://vindler.solutions/about"},"publisher":{"@type":"Organization","@id":"https://vindler.solutions/#organization","name":"Vindler","logo":{"@type":"ImageObject","url":"https://vindler.solutions/logo.png"}},"mainEntityOfPage":{"@type":"WebPage","@id":"https://vindler.solutions/blog/aeo-best-practices-ai-crawler-checks"}}
```
