A Standards-Grounded Method for Evaluating Website Agent-Readiness

Introduction Autonomous AI agents and generative-search crawlers increasingly retrieve, parse, and act upon Web resources with no human present to interpret a page's visual layout. Such a consumer is, for the duration of a retrieval, a finite-state machine: it issues HTTP requests, follows redirects, reads response headers, honors (or is refused by) a robots exclusion policy, parses structured data, and either succeeds or fails on each step against rules that are already written down. These rules live in published specifications. Whether an agent can discover a site's machine-readable description, resolve its entities to authoritative identifiers, ingest its content efficiently, and complete a secure transport handshake are all questions with deterministic, verifiable answers. They are not questions of marketing reputation, and no opaque "authority score" can answer them. Despite this, the prevailing way sites are assessed for "AI visibility" relies on proprietary, non-reproducible metrics whose inputs are undisclosed and whose results change without an observable cause. That approach is unfalsifiable and therefore unhelpful to an operator who wants to know, concretely, what to change. This document takes the opposite stance. It treats agent-readiness the way the W3C Markup Validation Service treats markup and the way public TLS test services treat a transport configuration: as a set of binary or bounded, independently reproducible observations against published specifications. If a server does not advertise a given link relation, or does not negotiate a given TLS version, that is a fact an auditor and the site operator can both confirm with a single request, not a matter of opinion. The method defined here rests on four evaluation pillars, each anchored to published standards. Agent Discoverability () concerns whether an agent can find a site's machine-readable surfaces; it draws on the Robots Exclusion Protocol for crawl authorization, on Web Linking for typed relations such as "describedby" and "alternate", on the api-catalog well-known URI and its application/linkset+json representation , and on the generic "/.well-known/" path prefix within which such resources are located. Semantic Entity Grounding () concerns whether a site expresses, in machine-readable form, what its entities are and to which authoritative identifiers they correspond; it draws on W3C JSON-LD 1.1 and the Schema.org vocabulary , and on the URI generic syntax for the requirement that cross-references resolve as absolute URIs. Content Efficiency and Ingestion () concerns whether an agent can obtain a site's content in a low-overhead, high-signal form; it draws on the text/markdown media type and on Web Linking . Transport () concerns whether delivery is secure and efficient for an automated consumer; it draws on HTTP Semantics , HTTP Caching , and TLS 1.3 . From the observations in these pillars the method computes a single deterministic score in the range 0 to 100 as a weighted sum, with the four pillars weighted 30, 30, 25, and 15 respectively. These weights and the formula that combines them are a reproducible convention defined by this document to aid comparison and prioritization. The convention is not itself a standard, and nothing in this document elevates the score, its weights, or any threshold to normative status. An implementation computing a different aggregate from the same per-pillar observations remains consistent with the underlying specifications; only the labeling of such an aggregate as the score defined here is constrained. The scope of this document is deliberately narrow. It specifies a measurement and diagnostic method and nothing more. It defines no new protocol, media type, link relation, header field, or well-known URI, and it changes the behavior of none that exist. It imposes no requirement on any site to adopt any practice. It is observation, not enforcement: an agent-readiness evaluation reports how a target behaves with respect to specifications that already define that behavior, and the decision to act on such a report rests entirely with the site operator. Two mechanisms that appear in the agent-readiness landscape are explicitly outside the set of standards this method relies upon. The "llms.txt" file is a community convention, not an IETF or W3C standard. Cloudflare's "Content-Signal" is a vendor mechanism, not a standard. Where present, these are observed only as supplementary signals and are reported as conventions, so that a site is never credited or penalized under this method merely for using, or not using, a particular vendor or community practice. A running reference implementation of this method, "Agent Authority" , exists. It captures the telemetry described above over live HTTP and computes the four-pillar score as a pure function of that telemetry, with an accompanying automated test suite, and is referenced in this document as running code demonstrating that the method is implementable and reproducible.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. In this document such requirements-language keywords describe the evaluation method itself (for example, what an implementation of this method is to observe and report) and the operational safety of performing an evaluation. They impose no conformance requirement on any evaluated site and no requirement on any protocol.

Evaluation Pillars and Standards Mapping This section defines the four evaluation pillars that constitute the agent-readiness method. Each pillar describes a class of observable facts about a target origin, each such fact is tied to the published specification that governs it, and each is checked by the reference implementation described in ("Agent Authority"). The method is diagnostic: it observes what an origin serves and reports it against the relevant standard. It does not define new protocol behavior, and nothing in this section places a requirement on any server operator. A note on terminology used throughout this section. Where a behavior is governed by a published IETF or W3C specification, this document cites that specification and treats the observation as conformance evidence. Where a behavior follows only a community or vendor convention with no corresponding standards-track or W3C Recommendation status, this document says so explicitly and treats the observation as an advisory signal with no conformance meaning. Two such conventions appear below: the "llms.txt" file and Cloudflare's "Content-Signal". Neither is an IETF standard or a W3C Recommendation. They are observed because agent operators encounter them in the wild, but they are never treated as standards compliance, and the reference implementation scores them only as minor observational bonuses so that a site is not rewarded merely for using a particular vendor.

Pillar One: Agent Discoverability This pillar observes whether an origin advertises machine-discoverable entry points that an autonomous agent can follow without guessing. Three independent observations are made against the final host (after redirects are followed and capped). The first observation is the robots exclusion policy. The reference implementation fetches /robots.txt and evaluates it using a parser that follows the grouping and longest-match semantics of the Robots Exclusion Protocol : records are grouped by case-insensitive user-agent token, the longest matching Allow or Disallow rule wins, and an Allow of equal specificity takes precedence over a Disallow. It then asks, for a representative set of agent user-agent product tokens, whether crawling of the site root is permitted. The governing standard is , which standardizes the file's location, syntax, and matching rules. What is observed is strictly the stated policy for named agent tokens; the method does not fetch disallowed paths and does not treat a Disallow as a defect, because a Disallow is a legitimate operator choice. The observation reported is simply whether, and for which agents, access is permitted. The second observation is a typed link from the root resource to a machine-oriented description of the site. The reference implementation parses the Link header field of the root response according to the Web Linking grammar and looks for a link whose relation type is "describedby". The "describedby" relation is an IANA-registered link relation (registered in 2009, with its defining reference being the W3C POWDER specification ) and indicates that the target resource is a description of the context resource. In the agent-readiness setting this is the idiomatic way to point an agent at a plain-text or structured site summary, including a file following the "llms.txt" convention. The governing standard for the link-serialization mechanism is , which defines the Link header field, the link model, and the registry of relation types; the "describedby" relation itself is governed by its IANA registration rather than by directly. The standard being satisfied is the Web Linking mechanism and the registered relation; the "llms.txt" file that the link may point to is a convention, not a standard, and the presence of the file is scored only as an observational signal under . The third observation is API discoverability via the api-catalog well-known resource. The reference implementation checks for a Link relation of type "api-catalog" on the root response and, independently, fetches /.well-known/api-catalog and inspects whether it is served with the application/linkset+json media type. Both the "api-catalog" link relation and the /.well-known/api-catalog well-known URI are defined by , "api-catalog: A Well-Known URI and Link Relation to Help Discovery of APIs". specifies that the catalog is represented as a link set and, in its primary representation, uses the application/linkset+json media type (the link set format is defined in ); the "api-catalog" relation type and the well-known URI are both registered by . This document is explicit and deliberate on one point of fact that is commonly gotten wrong: the api-catalog well-known URI is defined by and is not defined by . , "Well-Known Uniform Resource Identifiers (URIs)", defines only the generic "/.well-known/" path prefix and the procedures of the Well-Known URI registry; it does not define any specific well-known URI, and in particular it does not define api-catalog. The two are related only in that api-catalog is one entry registered under the general prefix that established. The method observes whether the api-catalog relation is advertised, whether the well-known resource is served, and whether its media type matches what prescribes. A closely related registered relation an origin may use for human-facing API documentation is "service-doc", registered by (Informational); the method may note its presence but treats catalog discovery through as the primary discoverability signal.

Pillar Two: Semantic Entity Grounding This pillar observes whether the origin publishes machine-readable structured data that identifies the entity the site represents and anchors it to external authorities, so that an agent can resolve the site to a known real-world entity rather than inferring one from prose. The reference implementation extracts every script element of type application/ld+json from the root document and performs a set of concrete, independently verifiable structural checks: whether each block is well-formed and parses as JSON; whether a node declares an @context that references the schema.org vocabulary; whether a node carries an @type; and whether any sameAs values are present and, if so, whether every one of them is an absolute URI. The governing specifications are the W3C JSON-LD 1.1 Recommendation for the serialization and processing model and the Schema.org vocabulary for the terms used (@type, sameAs, and so on). These are W3C and community-governed vocabularies and syntaxes rather than IETF RFCs, and this document cites them as such. The one IETF standard that applies directly here is , the generic URI syntax, which governs the absoluteness check applied to sameAs values: an entity cross-reference is only resolvable by an agent if it is an absolute URI (scheme and authority present) as defined by , and a relative reference cannot unambiguously identify an external entity. The method therefore observes the presence and structural shape of JSON-LD and the absolute-URI property of its sameAs anchors, citing for the latter. This document is deliberate about what is not claimed in this pillar. There is no authoritative, standardized validator that pronounces a given JSON-LD graph "valid" or "invalid" as a semantic whole; JSON-LD 1.1 defines well-formedness and processing, and Schema.org defines a vocabulary, but neither defines a single pass/fail validity verdict for an arbitrary entity description. For this reason the method reports concrete sub-observations (parses, references schema.org, has a type, has absolute sameAs anchors) and never an undefined "valid" boolean. This keeps every reported fact traceable to a specific, checkable property rather than to a subjective judgment.

Pillar Three: Content Efficiency and Ingestion This pillar observes whether the origin offers a representation of its content that is inexpensive for an agent to ingest, and whether it advertises that representation through standard mechanisms. The primary observation is a Markdown representation of the page. The reference implementation checks whether a Markdown mirror is served with the text/markdown media type and, independently, whether the root response advertises an alternate representation via a Link relation of type "alternate" carrying type="text/markdown". The media type text/markdown is registered by , "The text/markdown Media Type". Two facts about are load-bearing and this document states them precisely. First, is Informational; it is not a standards-track specification. Second, and critically, carries no notion of Markdown "validity": it cites the Markdown syntax specification's statement that there is no such thing as "invalid" Markdown. The method therefore never checks, scores, or reports Markdown "validity" or "correctness". What it observes is strictly the presence of a Markdown representation, the correctness of the media type label (text/markdown) under which it is served, and whether that representation is discoverable through a standard typed link. The "alternate" relation used to advertise the mirror is an IANA-registered link relation, and the Link header mechanism that carries it is governed by ; the absoluteness and syntax of the link target are governed by . A secondary observation in this pillar is a coarse text-to-markup density of the HTML body, reported only as an efficiency indicator for how much of the delivered payload is agent-usable text; this is an informational measurement, not a conformance claim against any specification. Within this pillar the method also records whether a file following the "llms.txt" convention is present at the origin. This document states plainly that "llms.txt" is a community convention, not an IETF standard and not a W3C Recommendation. It is observed because it is in common use as an agent-oriented site summary, but its presence is treated purely as an advisory signal. It carries no standards-conformance meaning, and the reference implementation scores it only as a small observational bonus, never as part of the standards-backed core of the pillar. Likewise, where an origin emits Cloudflare's "Content-Signal" directives (a vendor mechanism for expressing content-usage preferences), the method may record them as observational context only; "Content-Signal" is a Cloudflare vendor convention, not an IETF or W3C standard, and this document does not treat its presence or its directives as conformance to any standard.

Pillar Four: Transport This pillar observes properties of the transport layer that affect whether an agent can retrieve content securely, efficiently, and with cacheable results. All observations here are governed by standards-track IETF specifications. The first observation is the negotiated TLS version. The reference implementation reads the TLS protocol version from the live connection used to fetch the root resource and reports whether TLS 1.3 was negotiated. The governing standard is , which specifies TLS 1.3 (obsoleting RFC 8446). The method observes the negotiated version as a fact of the connection; because the version is only reliably readable from the open HTTP/1.1 network stream, the implementation makes this probe over HTTP/1.1, and where it cannot read the version it reports the observation as unknown rather than as a failure. The second observation is the HTTP caching policy. The reference implementation reads the Cache-Control header field of the root response and reports the declared caching policy, flagging internally contradictory directives (for example, a response that combines no-store with a positive max-age). The governing standard is , HTTP Caching, which defines Cache-Control and the semantics of its directives. This document is deliberate in observing the stated caching policy rather than any transient cache outcome: a per-request HIT or MISS status is not scored, because it varies between requests and would make results non-reproducible. The method may additionally note, in a vendor-agnostic way, that a response passed through an edge cache (for example via an Age header field, whose freshness semantics are defined by , or via one of several vendor cache-status header fields), but it treats the specific HIT/MISS value as observational only. The third set of observations concerns general HTTP semantics and responsiveness. The reference implementation records the response status handling, header semantics, and a time-to-first-byte measurement for the root fetch. The governing standard is , HTTP Semantics, which defines the meaning of methods, status codes, and header fields that the method relies on when interpreting every response. Time-to-first-byte is reported as a transport responsiveness indicator relevant to agents that must fetch many resources; it is a measurement rather than a conformance verdict against any single clause, and (together with ) frames how the underlying request/response exchange is interpreted.

Scoring Methodology This section defines how the per-pillar observations of are combined into a single composite figure. The reader is cautioned at the outset that the composite score, its range, its band labels, and the pillar weights given below are a reproducible convention defined by this document. They are an evaluation and reporting aid. They are not themselves an Internet standard, they do not extend or amend any of the protocols cited in , and conformance to the referenced RFCs and W3C specifications in no way depends on them. A different evaluator is free to weight the pillars differently; what this document fixes is a single, transparent, deterministic convention so that two independent runs against the same observed target produce the same number. The composite score is a weighted sum over the four pillars, where each pillar score is a real number in the closed interval 0 to 100 computed from the per-check results of that pillar, and each pillar score is clamped to that interval before it is combined: The weights sum to exactly 1.00, so the composite Score is itself bounded to the range 0 to 100. An implementation SHOULD assert that the weights sum to unity at startup; the reference implementation () does so. The rationale for the distribution is editorial, not normative: discoverability and entity grounding are weighted most heavily because they most directly determine whether an autonomous agent can find the resource and resolve it to a known entity at all, whereas transport characteristics, while they affect ingestion cost and latency, rarely prevent ingestion outright and are therefore weighted least. Readers who disagree with this emphasis can recompute the composite with their own weights without changing any of the underlying observations; the pillar scores are reported individually precisely so that this is possible. Reproducibility is the central design constraint of this convention, and it imposes two rules that distinguish this method from a naive checker. First, the method scores stable policy, never transient per-request state. The clearest instance is HTTP caching. The value of a transient edge-cache status header such as "cf-cache-status: HIT" or "cf-cache-status: MISS" (or any equivalent vendor header) varies between otherwise identical requests depending on cache warmth and is therefore unsuitable as the basis of a reproducible score. Accordingly, the Transport pillar scores the declared caching policy, principally the "Cache-Control" header defined in , together with the vendor-agnostic presence of an edge cache. The HIT or MISS value itself is observed and reported for information but is explicitly not an input to the score. The same principle applies throughout: the evaluator credits what the origin durably declares, not what a single probe happened to encounter. Second, the two best-known community and vendor conventions in this space, the "llms.txt" file and Cloudflare's "Content-Signal" mechanism , are treated as observational bonuses only, not as scored backbone. As stated in , neither is an IETF or W3C standard: "llms.txt" is a community convention and "Content-Signal" is a single vendor's feature. Their presence is recorded and may contribute a small, bounded bonus within a pillar, but a target can reach the top band without either of them, and their absence is never reported as non-conformance. This keeps the convention from silently rewarding the customers of one content delivery network or the adopters of one unratified file format, and keeps the score anchored in the published standards of . A separate concern is how the method treats a target it could not actually observe. An evaluator must not conflate "we were not allowed to look" with "we looked and the target failed." When the root resource is unreachable, for example because the connection fails, times out, is met with a WAF or bot-management challenge, or returns an HTTP 403 (Forbidden) or comparable blocking status, the target is reported as "not observed" (an explicit "unreachable" result) rather than being assigned a low or zero score. A "not observed" result carries the reason for non-observation (the blocking status or transport error) and a distinct band; it is not scored as RFC non-conformance, and it MUST be visually and semantically distinguished in any report from a reachable target that was examined and found wanting. This distinction matters because many origins front their content with challenge pages that a conformance crawler cannot pass, and penalising such a site as if its underlying resources were non-conformant would be both inaccurate and unfair. Only a target that is genuinely reachable and examined yields a pillar-by-pillar composite score. Finally, the composite number is a summary, not the substance of the result. The running reference implementation () emits, for every individual check within every pillar, one of three outcomes, "pass", "warn", or "fail" (with a fourth "unknown" outcome reserved for a signal that could not be measured on a given connection, such as a TLS version that the probe was unable to read), together with the specific clause being applied, the citation to the governing RFC or specification, the raw observation, and, where relevant, a suggested remediation and a command to reproduce the observation. The intent is that the composite Score lets a reader rank targets at a glance, while the per-check "pass/warn/fail" breakdown, each item citing the standard it rests on, lets that same reader audit exactly how the number was reached and verify it independently. The score is thus always presented alongside, and never in place of, the itemised evidence from which this document's convention derived it.

Security Considerations This document defines an evaluation method; it specifies no new protocol, defines no new wire format, and changes the behavior of no existing protocol. It is a measurement and diagnostic technique, not an enforcement mechanism. The security considerations below therefore concern the act of performing the evaluation and the interpretation of its results, rather than any new protocol machinery. The general guidance of on writing security considerations informs this section.

Active Probing and Authorization An agent-readiness evaluation as described here is an active-probe diagnostic. To gather the telemetry the four pillars require, an implementation sends real HTTP requests over TLS to a target origin and to a small, fixed set of resources beneath it, including the site root, "/robots.txt" , "/.well-known/api-catalog" , and other well-known or conventional paths under the "/.well-known/" prefix . These requests are made to third-party infrastructure that the party running the evaluation may not own or control. Operators MUST NOT treat the mere availability of this method as authorization to scan arbitrary hosts. An implementation and its operator are responsible for ensuring they have authorization to probe a given target, for example because they own or administer it, because its operator has consented, or because applicable law and the target's own published policies permit such requests. In the common case, an operator SHOULD restrict evaluation to sites they own or to sites whose operators have explicitly asked to be evaluated. Even where probing is authorized, implementations SHOULD send requests at a modest rate, SHOULD identify themselves honestly with a descriptive User-Agent rather than impersonating a browser or another crawler, and SHOULD honor the target's Robots Exclusion Protocol directives for the identity under which they fetch. Because a single evaluation issues only a few requests against well-known paths, its load on a target is small; nonetheless, operators running evaluations in bulk SHOULD apply rate limiting and result caching so that repeated or batched evaluations do not aggregate into abusive traffic against any one origin.

Server-Side Request Forgery A particular risk arises when an implementation of this method is exposed as a hosted service that accepts a target host or URL from an untrusted caller and then fetches it. Such a service is, by construction, a request-forwarding proxy, and without defensive measures it is vulnerable to Server-Side Request Forgery (SSRF): an attacker can supply a hostname that causes the service to issue requests to resources that are reachable from the service's own network position but not from the attacker's, such as internal services, management endpoints, or a cloud provider's instance metadata endpoint. This is a well-known class of vulnerability and is not specific to this document, but any hosted evaluation service inherits it. Implementations that accept caller-supplied targets MUST defend against SSRF. The critical property is to resolve-then-check: the supplied name MUST be resolved to its IP address or addresses, and each resulting address MUST be validated against a deny policy before any request is issued, because a public DNS name can resolve to a private, internal, or otherwise sensitive address, and that mapping can change between checks. The deny policy MUST reject, at minimum, loopback addresses, private-use addresses (the ranges described in and IPv6 unique-local addresses), link-local addresses (notably the IPv4 "169.254.0.0/16" range, which includes the "169.254.169.254" cloud metadata address, and the IPv6 "fe80::/10" range), multicast, reserved, and unspecified addresses, and in general any address that is not globally routable. Because a hostname can resolve to multiple addresses, every resolved address MUST pass the policy, not merely the first; a target that resolves to any denied address MUST be refused. Implementations SHOULD additionally constrain the scheme to HTTPS, cap the number and depth of redirects they will follow and re-apply the address check to redirect targets, cap response body sizes before parsing, and apply per-request timeouts, so that a hostile or pathological target cannot exhaust the service's resources or be used to pivot via a redirect to a denied address. The reference implementation referred to in this document ("Agent Authority") performs exactly this resolve-then-check validation before every scan and refuses any target that resolves to a non-public address; operators of other implementations are strongly encouraged to do likewise.

Interpretation of the Score The composite score defined by this document is informational. It is a reproducible convention for summarizing a set of observations about how amenable a site is to ingestion by autonomous agents and generative-search crawlers; it is not a security rating, a vulnerability score, a trust signal, or a certification, and it MUST NOT be used as, or as an input to, a security control, an access-control decision, an authentication or authorization mechanism, or an allow/deny determination. A high score indicates only that certain discoverability, structured-data, content-efficiency, and transport features were observed at scan time; it says nothing about the correctness, authenticity, safety, or trustworthiness of the site's content, and a site can be fully "agent-ready" by this method while serving hostile, deceptive, or incorrect content. Conversely, a low score reflects the absence or non-observability of those features, not that a site is malicious or non-conformant with any standard. Consumers of a score, whether human or automated, SHOULD treat it as a diagnostic summary to guide remediation, not as an assertion about security posture, and SHOULD NOT make trust decisions on the basis of it.

Observational Signals and Scan-Time Validity Several inputs to the method are point-in-time observations that reflect only what was true at the moment of the scan and from the vantage point of the scanner. In particular, any TLS-related observation, such as a report that a connection negotiated TLS 1.3 , records only the protocol version that happened to be negotiated for that one connection, from that client, at that time; it is not a statement about the server's full configuration, its support for other versions or cipher suites, the validity of its certificate chain, or its resistance to downgrade, and it can differ between clients and change at any time. Likewise, transport and caching observations describe the responses actually received and may vary with the request path, the requesting client, intermediary caches, content distribution networks, or the time of day. Results can also be affected by a target that serves a bot-challenge or interstitial page instead of its real content, which an implementation may be unable to distinguish from the genuine site; such results reflect what was observed, not necessarily the site's steady-state behavior. Finally, two of the signals the method may observe are community or vendor conventions rather than standards. The "llms.txt" convention is a community practice, not an IETF or W3C standard, and the "Content-Signal" mechanism is a vendor (Cloudflare) convention, not a standard. This document treats both as purely observational signals and does not normatively depend on them; implementations SHOULD label them as conventions wherever they are reported, so that consumers do not mistake their presence or absence for conformance to any published specification.

IANA Considerations This document has no IANA actions. This document defines an evaluation method that is grounded entirely in existing, already-registered standards and registry entries, and it introduces nothing that requires registration. In particular, the link relation types it observes, "describedby", "alternate", "api-catalog", and "service-doc", are all already present in the IANA "Link Relation Types" registry (see for the registry, for "api-catalog", and for "service-doc"), and the well-known URIs it observes, including the "api-catalog" well-known URI , are registered in the IANA "Well-Known URIs" registry established by . This document requests no additions to, deletions from, or modifications of any IANA registry, defines no new media types, link relations, well-known URIs, or protocol parameters, and establishes no new registry.