Crawlers & AI
What search engines and AI agents read on your site
Search engines, AI assistants and agents read your site differently than people do. I check what they see, what they may take, and how your site can offer useful data on purpose.
When it is time
Sounds familiar?
- AI answers rarely mention your site, or get the facts wrong
- You want to decide which AI crawlers may use your content
- Structured data is missing, or full of plugin defaults
- AI agents should work with your WordPress, but safely
The audit
What I check.
Remote and read-only. You get a written report with every finding, its risk and the effort to fix it.
- robots.txt, sitemaps and how crawlers are steered
- Rules for AI crawlers: which ones you allow, which ones you block
- llms.txt: a short guide to your site for language models
- Structured data (schema.org), and whether it matches the visible content
- Titles, descriptions and canonical URLs, also across languages
- Content that only appears with JavaScript
- WordPress abilities and MCP: what agents could do on your site, with which permissions
You can start with the free check, which already tests robots.txt with its sitemap and whether an llms.txt exists. It takes a few seconds and needs no access.
The fix sprint
What a sprint typically closes.
Fixed scope and price, agreed before we start. Every change arrives as a pull request that you review and merge.
- A robots.txt and AI crawler policy that reflects your decision
- An llms.txt per site, maintained in WordPress
- Clean structured data for your key page types
- Abilities for agents: read-only first, with clear permissions
FAQ
Questions I get asked.
Does an llms.txt guarantee visibility in AI answers?
No. It is a proposed standard, and not every AI system reads it. It is cheap to provide and helps the systems that do.
Should we block AI crawlers?
That is a business decision more than a technical one. I show you who crawls your site and what each option means. Then you decide.
What are WordPress abilities?
An API in WordPress Core that describes what a site can do, in a form AI agents understand. The free check on this site is one, so I know the details first-hand.
Tell me about your site.
A few lines are enough: what the site does, where it is hosted and what bothers you about it. If you are a web designer asking on behalf of a client, just say so. I reply personally, and if I think you don’t need an audit, I will say so.