What do schema, llms.txt and AI crawlers actually do?

Section 01

1. Structured data (schema / JSON-LD)

This is the chapter where most owners hand it over, and honestly that's a rational choice. Read it anyway, so you know whether whoever you're paying actually knows what they're doing.

A small block of code that labels your facts: this is the price, this is the phone number, this is the opening time. Be precise about what it does. Google has stated no special schema markup is required to appear in its generative results — but Product, Offer, AggregateRating, LocalBusiness and FAQ markup still power rich results, Shopping listings and the Shopping Graph, and structured product data is the single biggest lever for AI shopping surfaces. So: not a magic citation button, absolutely still infrastructure.

Ask your web person for this, verbatim:

"Please add valid JSON-LD for Organization and LocalBusiness sitewide, Service on each service page, Product + Offer + AggregateRating on products, and FAQPage where we have real FAQs. Then validate every template in Google's Rich Results Test and tell me which pass."

Section 02

2. Let the right robots in

Check robots.txt for blocks on GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended. Plenty of sites blocked these in a panic during 2024 and never told the marketing team. Decide deliberately: blocking training crawlers is a legitimate choice, but blocking the search crawlers is choosing to be invisible. Also make sure your key pages don't need JavaScript to render their text, and that your sitemap is current.

Section 03

3. llms.txt — the honest verdict

A plain text file at your root listing your important pages, like a menu for machines. It costs an hour. Here's the balanced truth: Google Search said in May 2026 that you don't need it for AI Overviews or AI Mode, and Google doesn't read it. It does do real work as a routing map for coding and research agents, it feeds Chrome's Lighthouse agentic browsing audit, and it's cheap insurance for a web where more browsing is done by software. Do it after everything else in this chapter, not instead of it.

Section 04

If that chapter made your eye twitch

Perfectly normal response. This is the exact work we do for people who'd rather run their business — audits, schema, crawler config and entity clean-up. The audit is £595 (credited in full against implementation) and implementation is from £2,400; the full list is at aeo-rex.com/pricing. The free tools will tell you first whether you even need us.