Crawlers & AI
Readable for search engines and AI agents. On your terms.
Search engines, AI assistants and agents read your site differently than people do. I check what they see, what they may take, and how your site can offer useful data on purpose.
When it is time
Sounds familiar?
- AI answers rarely mention your site, or get the facts wrong
- You want to decide which AI crawlers may use your content
- Structured data is missing, or full of plugin defaults
- AI agents should work with your WordPress, but safely
The audit
What I check.
Remote and read-only. You get a written report with every finding, its risk and the effort to fix it.
- robots.txt, sitemaps and how crawlers are steered
- Rules for AI crawlers: which ones you allow, which ones you block
- llms.txt: a short guide to your site for language models
- Structured data (schema.org), and whether it matches the visible content
- Titles, descriptions and canonical URLs, also across languages
- Content that only appears with JavaScript
- WordPress abilities and MCP: what agents could do on your site, with which permissions
Start with the free check: it already tests robots.txt with its sitemap and whether an llms.txt exists. Seconds, no access needed.
The fix sprint
What a sprint typically closes.
Fixed scope and price, agreed before we start. Every change arrives as a pull request that you review and merge.
- A robots.txt and AI crawler policy that reflects your decision
- An llms.txt per site, maintained in WordPress
- Clean structured data for your key page types
- Abilities for agents: read-only first, with clear permissions
FAQ
Questions I get asked.
Does an llms.txt guarantee visibility in AI answers?
No. It is a proposed standard, and not every AI system reads it. It is cheap to provide and helps the systems that do.
Should we block AI crawlers?
That is a business decision, not a technical one. I show you who crawls your site and what each option means. Then you decide.
What are WordPress abilities?
An API in WordPress Core that describes what a site can do, in a form AI agents understand. The free check on this site is one, so I know the details first-hand.
Let’s make your WordPress boring. In the best way.
Tell me about your site. You get an honest answer, even if it is “you don’t need me”.