The definitive guide to optimizing your website's AEO

The intermediary changed. It is no longer a list of blue links: it is a language model that reads, summarizes and decides whom to cite. Optimizing for that model has its own discipline, and here it is in full.

This guide was built from the real optimization program Transgenia ran on transgenia.org between March and July 2026. Every example is a measure we applied and verified in production, not theory.

0. What AEO is (and why SEO alone is no longer enough)

SEO (Search Engine Optimization) optimizes to get a human to click a blue link in a list of results. AEO (Answer Engine Optimization) and its cousin GEO (Generative Engine Optimization) optimize to get an answer engine (ChatGPT, Claude, Perplexity, Google AI Overviews, Bing Copilot) to understand, trust and cite your content when it generates a response.

The underlying shift is this: the intermediary is no longer a SERP, it is a language model. That model does not "click". It reads, summarizes, decides what counts as a trustworthy source, and chooses whom to attribute. Optimizing for AEO means making your site machine-readable, structured as knowledge, and demonstrably trustworthy.

The thesis of this guide, and Transgenia's bet as an AI boutique, is simple. The most powerful form of AEO is not describing what you do. It is making your own site run governed AI that engines can observe and cite. The cobbler wearing good shoes.

Mermaid meta-structure

flowchart TB
    subgraph N4["Level 4 · Live AI demonstration (differentiator)"]
        D["RAG concierge with citations · governed · LFPDPPP"]
    end
    subgraph N3["Level 3 · Citeable entity"]
        C["llms.txt · JSON-LD Organization/knowsAbout · FAQPage · es/en/zh consistency"]
    end
    subgraph N2["Level 2 · Structured content"]
        B["Q&A / FAQ · single source faq.json · cornerstone blog · feeds"]
    end
    subgraph N1["Level 1 · Technical foundations"]
        A["Crawlability for AI bots · performance (LCP) · valid structured data"]
    end
    N1 --> N2 --> N3 --> N4
    style N4 fill:#E14228,color:#fff
    style N3 fill:#FB6C25,color:#fff
    style N2 fill:#FBA225,color:#101820
    style N1 fill:#22A088,color:#fff

1. Level 1: technical foundations

An AI engine cannot cite what it cannot crawl, or what loads so slowly that its crawler gives up.

1.1 Crawlability for AI bots (the step almost everyone skips)

robots.txt has to explicitly allow the AI crawlers, not just Googlebot. At Transgenia we enabled, by name: GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, bingbot, plus the Chinese search engines (Baiduspider, Sogou, 360Spider, YisouSpider, Bytespider) for the ZH market.

Expensive gotcha: on Cloudflare, the edge "Managed Content Signals" block can override your robots.txt Allow rules and block AI bots even though your file says otherwise. If your goal is to be cited, disable it in the dashboard. The file alone is not enough.

Two surgical rules we applied: Disallow: /cdn-cgi/ (a Cloudflare internal endpoint that returned 4xx and polluted Bing WMT) and Disallow: /assets/docs/ (privacy-notice .docx files that should not burn crawl budget). Everything else stays crawlable.

1.2 Performance: if it's slow, you won't be cited

We measured the real baseline with PageSpeed Insights, not third-party reports: mobile Perf 56, LCP 20.8 s. The root cause was a single one. The variable font Material Symbols Rounded weighed 5.2 MB (44% of the payload) and loaded on the critical path across 36 pages, when the entire site uses only 3 icons.

Fix: subset via &icon_names=auto_awesome,cloud_sync,inventory_2&display=block on the 3 pages that use them, and remove the <link> on the 33 that don't. Result: 5.2 MB to 3.7 KB (99.9%), verified in a real browser (the 18 px glyph, not the fallback text).

AEO lesson: performance is not cosmetic. A 20 s LCP on a slow network is a crawler that gives up before it reads your content.

1.3 Valid structured data

JSON-LD Organization, WebSite, WebPage, Service, BreadcrumbList and, key for AEO, FAQPage. Hard rule: structured data must match visible content on the page, or Google ignores it (and may penalize). That is why we generated the FAQPage alongside a visible FAQ section, from a single source (§2.3).

2. Level 2: content structured as knowledge

2.1 llms.txt: the direct pitch to the machines

llms.txt is a root file designed so LLMs grasp your entity at a glance. Transgenia's used to describe you as an "Industry 4.0 consultancy, Odoo its core practice", which sabotaged the repositioning and told engines something different from what the company is today. We rewrote it to lead with "technology boutique specialized in applied AI", reordering the services (AI first) and adding an explicit note to AI agents about the positioning and where each fact lives.

2.2 FAQPage where it actually matters

The AI page (soluciones-ia.html) had no FAQPage, and it is the page where it helps most for an engine to cite answers. We added a visible FAQ plus FAQPage JSON-LD in es/en/zh, with Q&A grounded in real content (Claude workflows in production, measurement with observability, a 6-week pilot, responsible AI plus LFPDPPP, Odoo/WhatsApp integration). Zero fabrication.

2.3 Single source: faq.json

Instead of duplicating questions across HTML, JSON-LD and i18n (they drift out of sync), we created assets/data/faq.json as the single source that feeds both the FAQPage and the concierge's RAG index. Today it holds 5 sections (AI, about us, clinics, trading companies) with 54 knowledge points, all verified by grep against the real pages.

Mermaid meta-structure

flowchart LR
    subgraph SEO["Classic SEO"]
        K["Keywords"] --> L["Backlinks"] --> S["SERP position"] --> Click["Human click"]
    end
    subgraph AEO["AEO / GEO"]
        E["Entities & facts"] --> Q["Structured Q&A content"] --> T["Trust / evidence"] --> Cita["Citation in the AI answer"]
    end
    SEO -. evolves .-> AEO
    style SEO fill:#eef,stroke:#88a
    style AEO fill:#E14228,color:#fff

3. Level 3: being a coherent, citeable entity

3.1 Message consistency (humans and machines read the same thing)

We unified the value proposition to "Technology beyond production · Technology boutique" across 45 surfaces (hero, meta tags, og/twitter, about, JSON-LD knowsAbout) in es/en/zh. knowsAbout now leads with Artificial Intelligence, generative AI, RAG, governed AI agents. An engine that cross-checks your home, your about, your llms.txt and your schema has to read the same identity in all of them.

Trilingual gotcha: on this site EN/ZH have the copy embedded in the page, while ES is overwritten by i18n at runtime, so you must edit both sides or the change reverts.

3.2 Indexing hygiene (GSC + Bing WMT)

We curated the findings from Google Search Console and Bing Webmaster Tools: titles over 70 chars shortened; titles too short extended to 50–60; missing H1 added; the 4xx on /cdn-cgi/l/email-protection resolved with Disallow. Telling signal from noise matters. "Page with redirect" (http to https, www to apex canonicalization) needs no action; "discovered, not indexed" is resolved by requesting indexing, not by touching code.

Mermaid meta-structure

sequenceDiagram
    participant B as AI bot (GPTBot/ClaudeBot)
    participant R as robots.txt / Cloudflare edge
    participant P as Page (HTML + JSON-LD + FAQPage)
    participant L as llms.txt
    participant M as Answer engine
    B->>R: May I crawl?
    R-->>B: Allow (AI bots enabled, Managed Content OFF)
    B->>L: Read entity (who Transgenia is)
    B->>P: Read content + structured data
    P-->>B: FAQPage Q&A + Service + Organization
    B->>M: Index entity + facts + sources
    Note over M: User asks "who does Odoo+AI in LATAM?"
    M-->>M: Retrieve citeable facts
    M->>P: Cite transgenia.org as a source

4. Level 4: making your site demonstrate AI (the differentiator)

Here is the move that separates an AI boutique from everyone who only claims it: an AI concierge on the site itself, with RAG over your content, answering with cited sources, governed and privacy-respecting. It is three things at once: a live demo of your capability, conversational content that reinforces your entity, and a lead-capture machine.

4.1 Architecture (the PII stays in Mexico, LFPDPPP)

Mermaid meta-structure

flowchart TB
    U["Visitor (browser)"]
    subgraph EDGE["Cloudflare (edge)"]
        W["Worker /api/concierge/*"]
        KV["(KV: limits, quotas, blocks)"]
    end
    subgraph VPS["Mexico VPS (LFPDPPP)"]
        PB["PocketBase · registration + leads (PII)"]
        NGINX["nginx /pb/*"]
    end
    subgraph CLOUD["Services"]
        QD["(Qdrant Cloud · vectors)"]
        OAI["OpenAI · embeddings + gpt-4.1-mini"]
    end
    U -- "registration (Turnstile)" --> NGINX --> PB
    PB -- "session JWT (opaque sid)" --> U
    U -- "chat {sid, message}" --> W
    W <--> KV
    W -- "embed + retrieve" --> QD
    W -- "generate with citations" --> OAI
    W -- "opaque log (no PII)" --> NGINX
    style VPS fill:#22A088,color:#fff
    style EDGE fill:#FB6C25,color:#fff

Key privacy decision: the PII (first name, last name, email, phone) only lives in PocketBase, on the VPS in Mexico. Registration issues a JWT with an opaque id (sid); the Worker and OpenAI (US) only ever see {sid, query text}. Zero contact data crosses the border. This isn't only LFPDPPP compliance. It is a trust signal that E-E-A-T and the engines reward.

4.2 The RAG pipeline (why answers are citeable, not invented)

Mermaid meta-structure

flowchart LR
    subgraph INGEST["Ingestion (build)"]
        SRC["blog-src/*.md + faq.json"] --> CH["Chunking ~1200c"] --> EM1["Embeddings text-embedding-3-small"] --> UP["Upsert → Qdrant (397 pts)"]
    end
    subgraph RUNTIME["Runtime (per query)"]
        Q["User query"] --> EM2["Embed query"] --> SR["Top-k search in Qdrant"]
        SR --> TH{"score ≥ 0.35?"}
        TH -- no --> NF["'Not in my docs' + CTA (no LLM call)"]
        TH -- yes --> GEN["gpt-4.1-mini with context"] --> CIT["Streamed answer + citations"]
    end
    UP -. index .-> SR
    style NF fill:#FBA225,color:#101820
    style CIT fill:#22A088,color:#fff

The principle that protects your reputation is zero-fabrication. If retrieval does not clear the threshold, the concierge does not call the model. It answers "I don't have that documented, book a call". It cites the source page in every answer. That is exactly the discipline AI engines value.

4.3 Governance: limits, anti-abuse and budget

Mermaid meta-structure

flowchart TB
    REQ["Chat request"] --> IPB{"IP blocked?"}
    IPB -- yes --> BLOCK["403 + notify-admin screen"]
    IPB -- no --> JWT{"Valid session JWT?"}
    JWT -- no --> R401["401"]
    JWT -- yes --> INJ{"Prompt injection?"}
    INJ -- yes --> RJ["422 + strike (3 → IP block)"]
    INJ -- no --> RLM{"≤ 3/min and ≤ 15/week?"}
    RLM -- no --> R429["429 + call CTA"]
    RLM -- yes --> BUD{"Daily budget OK?"}
    BUD -- no --> R503["503 (kill-switch)"]
    BUD -- yes --> RAG["RAG + generation"]
    style BLOCK fill:#E14228,color:#fff
    style RAG fill:#22A088,color:#fff

Humanity is guaranteed with Turnstile at registration; the anti-abuse criteria (registration-farming by phone/IP, disposable emails with MX check, bursts, prompt-injection) each live where they belong.

4.4 Decision matrix: value first, sales-oriented

Mermaid meta-structure

flowchart TB
    IN["User message"] --> CLASS["Classify: service line + vertical + buying stage"]
    CLASS --> VALUE["Deliver real, grounded, CITED value"]
    VALUE --> QUAL["Qualify (BANT-lite: need, urgency, data, fit)"]
    QUAL --> HIGH{"High intent?"}
    HIGH -- yes --> HUMAN["Escalate to human (Calendly / quote)"]
    HIGH -- no --> CTA["Guide to the right CTA (page / estimator / demo)"]
    style VALUE fill:#22A088,color:#fff
    style HUMAN fill:#E14228,color:#fff

4.5 Controlled web search (fallback)

When the site's own RAG isn't enough and the topic is on-topic, the concierge can use web search (the Responses API web_search tool, same API key, zero new secrets) with a deterministic gate, caps (3/week/sid, 15/day), anti-injection (web content is data, never instruction) and one hard rule: it never answers about Transgenia using the web. That comes only from the site's own content. Default OFF, enabled via a variable.

5. Cross-cutting level: trust and E-E-A-T for machines

AEO rewards the same things as E-E-A-T, only machine-readable: evidence, governance and honesty. What Transgenia makes observable:

• Zero-fabrication in the concierge (it cites, or it admits it doesn't know).
• Demonstrable privacy (PII in Mexico, minimal transfer declared).
• Sourced content (real cases, not invented testimonials; we ruled out fabricating social proof).
• A cornerstone blog on governed AI: the philosophy written down, which the concierge also embodies.

6. Measurement (no AEO without evidence)

Golden rule (Definition of GREEN): never declare something "done" by verifying the artifact (the code, the commit). Only the observable effect counts: the curl to production, the entry in the index, the lead in the database.

Mermaid meta-structure

flowchart LR
    C["Change"] --> D["Deploy"] --> OBS{"Observable effect?"}
    OBS -- "yes (curl/index/lead)" --> GREEN["GREEN ✓ (document + diary)"]
    OBS -- "no" --> PEND["PENDING-VERIFICATION (what's missing, who, deadline)"]
    PEND --> C
    style GREEN fill:#22A088,color:#fff
    style PEND fill:#FBA225,color:#101820

7. The program, over time (what Transgenia executed)

Mermaid meta-structure

flowchart LR
    subgraph F["Foundations · Jul 17-18"]
      A1["GSC / Bing curation + robots"] --> A2["Perf: Material Symbols 5.2MB→3.7KB"]
    end
    subgraph E["Entity & content · Jul 18-19"]
      B1["Repositioning es/en/zh"] --> B2["llms.txt + JSON-LD + feed"] --> B3["faq.json + FAQPage"]
    end
    subgraph D["AI demonstration · Jul 19-21"]
      C1["Retire Odoo chat"] --> C2["Concierge backend (Worker + RAG)"] --> C3["Widget + deploy + GREEN prod"] --> C4["AI estimator + web search"]
    end
    F --> E --> D
    style F fill:#22A088,color:#fff
    style E fill:#FB6C25,color:#fff
    style D fill:#E14228,color:#fff

8. Actionable playbook (checklist)

Foundations

• [ ] robots.txt allows AI bots by name; Cloudflare Managed Content disabled.
• [ ] Mobile LCP under 2.5 s; no heavy fonts or images on the critical path.
• [ ] Valid JSON-LD (Rich Results) matching visible content.

Entity and content

• [ ] llms.txt describes your real entity, leading with your differentiator.
• [ ] Visible FAQPage on your key pages, from a single source.
• [ ] Identical identity across home, about, llms.txt and schema (all languages).

AI demonstration (advanced)

• [ ] A RAG assistant that cites sources and admits what it doesn't know (zero-fabrication).
• [ ] Demonstrable privacy (minimize and declare transfers).
• [ ] Governance: limits, anti-abuse and budget at the edge.

Measurement

• [ ] GSC + Bing WMT + Brand Radar + Lighthouse on a regular review cadence.
• [ ] Nothing is declared done without an observable effect (GREEN).

How it applies to your company

None of the above is exclusive to an AI boutique. The same discipline works for any company that decided to take its presence in front of answer engines seriously in 2026: technical foundations first, then a coherent, citeable entity, and finally, if you can, a live demonstration of what you do. At Transgenia we apply this to our own site before proposing it to anyone. To see how it translates to your industry, our AI Solutions page describes it by sector.

Frequently asked questions

What is the difference between SEO, GEO and AEO?
SEO optimizes to get you into a search engine's results list so someone clicks. GEO (Generative Engine Optimization) works to get a generative engine like ChatGPT or Perplexity to cite you as a source. AEO (Answer Engine Optimization) goes one step further: it works so the engine's answer delivers directly who you are or what you do, without forcing a click. In 2026 it pays to optimize all three, because buyers consult all three.

How do I let AI bots crawl my site?
In robots.txt, with explicit Allow rules by bot name (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, among others). Watch out for Cloudflare: its "Managed Content Signals" block can override those rules from the edge even if your file says otherwise. You have to check it in the dashboard, not only in the file.

Does structured data (JSON-LD) improve my AEO?
It helps, with one condition: the schema must match visible content on the page. A FAQPage with no real FAQ section can be ignored or penalized. Generate the JSON-LD from a single source so it never drifts from the text the user sees.

What is an llms.txt and what is it for?
It is a file at the site root designed so a language model grasps your entity at a glance: who you are, what you do, where each fact lives. It does not replace content, but it gives engines a clean summary led by your real differentiator.

Do I need a chatbot on my site to do AEO?
No, it is not mandatory. Levels 1 to 3 (technical foundations, structured content and a citeable entity) already move the needle. A RAG assistant that cites sources and admits what it doesn't know is level 4: the differentiator that demonstrates the capability instead of just claiming it. It is optional, but it is where an AI boutique separates from the rest.