An AI agent readable website gives software five things: an index it can find, content it can parse cheaply, stated rules for reuse, structured data it can query, and clear limits on what needs a login. We built that stack on EveryYarn, a Laravel yarn encyclopedia that Finarina LLC runs. This post walks through what the site serves today, the decisions behind each layer, and a Laravel pattern you can copy. It also says plainly what we cannot claim.
Everything below about EveryYarn comes from the live site, fetched on October 18, 2026. We did not measure traffic, citations, or rankings, so none appear here.
Why a Catalog Site Needed an Agent Surface
EveryYarn lists thousands of yarns, brands, fiber weights, fiber materials, and colorways. That is structured data that people and software both want. A person browses it. An agent usually wants one answer, such as the weight category of a specific yarn.
An agent that scrapes the HTML pays for navigation, scripts, and class names it will never use. It also guesses at structure that the site already knows. The fix is to publish the structure on purpose and to say which uses of the content are welcome.
That produces five layers. Each one has a public spec behind it, and each one fails differently when it is wrong.
What EveryYarn Serves Today
An llms.txt that indexes every machine door
The llms.txt proposal requires one thing: an H1 with the site name. It then allows a blockquote summary, free text, and H2 sections of links. EveryYarn’s file uses those H2 sections as a map: Site Structure, Content Types, AI/MCP Endpoints, Machine-Readable Specifications, Structured Data, and Contact.
The useful part is the endpoint list. It names each data URL with a one line description and the paging parameters. An agent that reads one file learns where every other door is. The file at everyyarn.com/llms.txt also links the OpenAPI document, the API catalog, the MCP server card, an auth guide, and the sitemap.
Compare that with a default generated file. Our WordPress llms.txt guide covers why a list of pages and categories is not enough. A curated file describes what the site is for and where the data lives.
Content signals in robots.txt
EveryYarn’s robots.txt carries one signal line among the usual rules (excerpt):
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Sitemap: https://www.everyyarn.com/sitemap.xmlCloudflare’s Content Signals Policy defines three terms. The search signal covers building a search index and returning links and excerpts, without AI-generated summaries. The ai-input signal covers feeding content into AI models. The ai-train signal covers training or fine-tuning models.
Our choice was yes, yes, no. We want to be found, and we want agents to answer questions from the catalog. We do not want it used as training data. A missing signal is neutral, not a refusal, so we state all three. The file’s own comments also say that the signals are formal reservations of rights under an EU copyright directive. A signal states a preference. It does not enforce one.
Markdown on request
When a client asks for a page as markdown, the server can return the same URL in that format. We covered the mechanics in our WordPress markdown content negotiation guide. On EveryYarn, a fetch tool that asked for markdown received the Brands page as markdown. The response began with YAML front matter that held the canonical URL (https://www.everyyarn.com/brands) and the title, followed by the page headings.
That front matter is a good design choice. It tells the agent which URL is canonical, so a citation points at the right place. We could not read the raw response headers from our audit environment, so we do not quote the Content-Type or Vary values here. Check those yourself with the script near the end of this post.
Open JSON endpoints for the catalog
The llms.txt file lists seven public data URLs under /mcp/, plus the index. The index at everyyarn.com/mcp describes six of them with their fields. The seventh is a short FAQ.
- Brands: name, URL, website, description, year founded, and yarn count. Paged.
- Catalog stats: total yarns, brands, weights, and materials. Not paged.
- Colorways: name, color family, yarn name, and brand name. Paged.
- Fiber materials: name, aliases, and yarn count. Not paged.
- Fiber weights: name, wraps per inch, and yarn count. Not paged.
- Yarns: name, brand, weight, fiber content, retail details, gauge, needle size, put ups, colorways, and a discontinued flag. Paged.
Paged endpoints accept page and per_page. The default page holds 100 items, and the maximum is 500. These endpoints need no token.
An MCP endpoint and a server card
The site also publishes an MCP endpoint at https://www.everyyarn.com/mcp. Its server card lists version 1.0.0, the streamable HTTP transport, and a tools capability with listChanged set to false. The server card gives clients one place to read how to connect.
The MCP transport specification sets the rules for that transport. The server exposes a single endpoint that handles POST and GET. Clients send each JSON-RPC message as a new POST and list both application/json and text/event-stream in the Accept header. The specification also says servers must validate the Origin header and should authenticate connections. Put those two checks on your own list.
A linked API catalog
The file at /.well-known/api-catalog follows RFC 9727. It links the OpenAPI document, llms.txt, and the MCP endpoint, so a client can discover all three from one known path. RFC 9727 expects the Linkset format with the application/linkset+json media type and a profile parameter. The links we read carry only an href each, so there is room to add relation types and a profile. Treat that as a refinement, not a flaw.
An authenticated API with explicit limits
Public read data and account actions live apart. The OpenAPI 3.1 document describes a bearer token scheme with scopes such as yarns:read, brands:read, and assessments:write. The auth guide says tokens identify an application, not a person. It also says registration is manual by email, anonymous access to those routes is not supported, and the default limit is 60 requests per minute. The server answers 401 without a valid token, 403 for a missing scope, and 429 over the limit.
That boundary matters. An agent should never wonder whether it may call a route. The docs say so before it tries.
Design Decisions That Transfer to Other Sites
Publish one index and keep it current
Every other layer depends on llms.txt being accurate. Generate it from the same source as your routes, or review it with each release. A stale index is worse than none, because agents trust it.
State your reuse rules before a crawler guesses
Pick all three content signals deliberately. A news publisher and a data catalog will choose differently. Write the reasoning down, because a client will ask why ai-train is set the way it is.
Separate public reads from authenticated writes
Reference data is cheap to serve and safe to share. Account data is neither. Keeping them on different routes lets you cache one aggressively and lock down the other.
Send Vary when one URL has two formats
If a URL returns HTML to browsers and markdown to agents, a shared cache needs Vary: Accept. Without it, one agent request can poison the cached page for every browser. Test through your real CDN, not only the origin.
A Laravel Pattern for Markdown Responses
This section is a reference pattern, not EveryYarn’s source code. It shows one clean way to add markdown negotiation to a Laravel 12 app that renders Blade views. Each HTML view gets a twin under resources/views/markdown/ with the same name. The middleware swaps the twin in when the client prefers markdown.
Laravel documents the handle(Request $request, Closure $next): Response signature, and it allows work after $next($request) returns. We use that to read the original view name from the response.
<?php
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\Request;
use Illuminate\View\View;
use Symfony\Component\HttpFoundation\Response;
class ServeMarkdown
{
public function handle(Request $request, Closure $next): Response
{
$response = $next($request);
// Tell shared caches that this URL varies by the Accept header.
$response->setVary('Accept', false);
if (! $request->isMethod('GET') || ! $this->wantsMarkdown($request)) {
return $response;
}
$original = method_exists($response, 'getOriginalContent')
? $response->getOriginalContent()
: null;
if (! $original instanceof View) {
return $response;
}
$twin = 'markdown.' . $original->name();
if (! view()->exists($twin)) {
return $response;
}
return response(view($twin, $original->getData()), 200)
->header('Content-Type', 'text/markdown; charset=utf-8')
->setVary('Accept', false);
}
private function wantsMarkdown(Request $request): bool
{
$accept = strtolower($request->header('Accept', ''));
if (! str_contains($accept, 'text/markdown')) {
return false;
}
$markdown = $this->weight($accept, 'text/markdown');
return $markdown > 0 && $markdown >= $this->weight($accept, 'text/html');
}
private function weight(string $accept, string $type): float
{
$main = strtok($type, '/');
$best = ['spec' => -1, 'q' => 0.0];
foreach (explode(',', $accept) as $range) {
$parts = array_map('trim', explode(';', $range));
$media = array_shift($parts);
$q = 1.0;
foreach ($parts as $part) {
if (str_starts_with($part, 'q=')) {
$q = (float) substr($part, 2);
}
}
$spec = match ($media) {
$type => 2,
$main . '/*' => 1,
'*/*' => 0,
default => -1,
};
if ($spec > $best['spec']) {
$best = ['spec' => $spec, 'q' => $q];
}
}
return $best['q'];
}
}The check requires a client to name text/markdown and rank it at least as high as text/html. A bare wildcard is no vote for markdown, so ordinary browsers keep getting HTML. The weight function ranks an exact type above a type wildcard, and a q=0 rejects the type.
Register the middleware on the web group in bootstrap/app.php:
use App\Http\Middleware\ServeMarkdown;
use Illuminate\Foundation\Configuration\Middleware;
->withMiddleware(function (Middleware $middleware): void {
$middleware->appendToGroup('web', [ServeMarkdown::class]);
})Laravel documents appendToGroup for this. Add the twin views next, write each as plain markdown, and keep front matter at the top with the canonical URL and title. Blade’s double brace syntax escapes HTML entities, so an ampersand arrives as & in your markdown. Use the unescaped syntax only for values you trust.
Audit Your Own Site
Run these checks against any site, including ours. Each one maps to a layer above.
#!/usr/bin/env bash
SITE="https://www.example.com"
# 1. Discovery files
curl -sS "$SITE/llms.txt" | head -40
curl -sS "$SITE/robots.txt" | grep -i "content-signal"
curl -sS -D - -o /dev/null "$SITE/.well-known/api-catalog"
# 2. Markdown on request, and HTML for everyone else
curl -sS -D - -o /dev/null -H "Accept: text/markdown" "$SITE/" | grep -iE "^(HTTP|content-type|vary)"
curl -sS -D - -o /dev/null -H "Accept: text/html" "$SITE/" | grep -iE "^(HTTP|content-type|vary)"
# 3. MCP initialize (headers required by the Streamable HTTP transport)
curl -sS -X POST "$SITE/mcp" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"audit","version":"1.0"}}}'Read the output for four things. The markdown request should return text/markdown; charset=utf-8 and a Vary header that includes Accept. The HTML request should still return HTML. The MCP initialize call should return JSON or an event stream, not a 4xx. The llms.txt file should list every endpoint the other checks touched.
What We Cannot Claim
We have no evidence that any of this raises search rankings or earns citations from AI products. We built it because the data is structured and agents are a real class of client. The goal is correct, cheap, well-labeled access. Anyone who promises a ranking lift from llms.txt or MCP should show the data.
Other gaps are known. Our audit environment blocked direct HTTP requests to the site, so we read pages through a fetch tool. We could not confirm response headers, and we could not run a live MCP session. The counts in llms.txt and the stats endpoint also differ slightly, which suggests one of them is cached. Until we fix that, we quote neither number.
Which Approach Fits Your Site
- A catalog or directory with structured records: publish read only JSON endpoints and an MCP endpoint, and document them in llms.txt.
- A content site on WordPress: start with a curated llms.txt and markdown negotiation, then decide your content signals.
- A Laravel application with accounts: add the public layers first, keep token scopes narrow, and document rate limits.
- Any site behind a CDN: test
Varybehavior at the edge before you announce anything.
Want this built or audited on your WordPress or Laravel site? Contact FKW Digital or book a consultation.
Leave a Reply