An Architecture for Open, Decentralized, and Scalable Large Language Model Inference

Introduction LLM inference is a stateful network service. A request selects a model and execution configuration, performs a prefill operation over the input, and then generates output tokens incrementally. Each generation step depends on model parameters and per-request state accumulated by earlier steps. Implementations commonly divide a model into stages and retain key-value (KV) caches near the stages that use them. Most distributed inference systems remain under one administrative entity. The entity admits resources, distributes model artifacts, selects execution nodes, monitors them, and decides which results to trust. The Intelligence Delivery Network (IDN) architecture deliberately uses this provider-controlled model and applies CATS concepts to the placement and selection of distributed inference capabilities. ODSI considers a different environment. Compute nodes belong to independent entities, no prior contractual trust is assumed, membership can change, and nodes may be faulty, rational, or malicious. A model can be represented by content-addressed artifacts held by multiple participants, and a request can be executed by a temporary path assembled from their resources. The objective is technical decentralization of control and custody, i.e., no single administrative entity is required to control participant admission, model-artifact availability, execution-path selection, result verification, or contribution settlement. Existing techniques in peer-to-peer content distribution and decentralized ledgers illustrate different parts of this design space. Content-addressed pieces, peer discovery, and replication are useful for distributing model artifacts . Permissionless identity, Sybil resistance, and consistent settlement state are relevant to resource contribution . But neither of them solves inference correctness. A digest proves the integrity of an artifact but not that a participant evaluated a model stage correctly, and repeating an expensive, numerically variable computation is not equivalent to validating a small deterministic ledger transition. This document therefore separates artifact integrity, execution correctness, service timeliness, and economic settlement. Each has different evidence and trust requirements.

Goals ODSI has the following architectural goals:

allow independently operated nodes to advertise partial resources and participate without relying on one service provider;
identify model artifacts, execution semantics, requests, steps, and state sufficiently precisely for independent participants to interoperate;
construct state-aware execution paths using heterogeneous and changing resource information;
attribute protocol actions and make the limits of that attribution explicit;
support selectable verification methods with stated correctness, cost, latency, and trust properties;
contain failures and permit recovery without requiring unrelated requests to stop; and
permit, but not require, a settlement mechanism that accounts for independently supplied resources.

Scope This document defines a protocol-independent architecture. It describes the information that crosses component boundaries and the properties that a complete deployment profile needs to provide. Separate specifications can define encodings, transports, discovery mechanisms, verification methods, consensus protocols, or settlement systems. ODSI is an application-layer overlay. The underlay forwards packets without knowledge of models, tensors, token positions, deadlines, rewards, or execution paths. An ODSI implementation can use existing secure transports and can use CATS inside a provider domain, but this document does not modify Internet routing or transport protocols.

Non-Goals This document does not:

standardize a token, ledger, consensus algorithm, proof system, trusted execution environment, or reputation formula;
assert that decentralization always reduces latency, cost, energy use, data movement, or concentration;
provide confidentiality merely by splitting a model or request among nodes;
define legal ownership or licensing of a model;
require that no participant ever stores a complete model; or
guarantee progress during an unbounded network partition.

ODSI can remove the technical requirement for a single operator, but it cannot by itself decentralize model ownership, licensing, governance, or economic power. A deployment therefore needs separate policies for model authorization, protocol changes, participant governance, and compensation, as discussed in .

Relationship to CATS and IDN CATS selects a suitable service contact instance using network and compute information in a single service-provider environment . IDN adds inference-specific capability, artifact, state, and policy concepts while retaining a provider-controlled administrative domain . ODSI differs in the trust and control model:

Property	IDN	ODSI
Administrative scope	One provider-controlled domain	Independently operated participants
Admission	Provider policy	Open discovery plus profile-selected admission and Sybil resistance
Control	Provider control plane	No required global controller
Artifact custody	Provider managed	Content-addressed, replicated, and potentially partitioned across participants
Execution trust	Provider assurance and policy	Explicit verification profile for mutually untrusted nodes
Accounting	Provider internal	Optional cross-entity settlement
Failure model	Managed faults	Faults, churn, equivocation, collusion, and resource attacks

ODSI does not treat the global peer overlay as a CATS domain. A participant that controls multiple sites can use CATS to select a local resource, and CATS metrics can inform that local decision. ODSI then treats the selected endpoint as one participant in its overlay. CATS does not provide ODSI peer admission, cross-domain trust, model governance, execution proofs, or settlement.

Terminology The terms below have architectural meaning in this document.

ARTIFACT: A content-addressed object used for inference, such as model weights, a tokenizer, an adapter, executable code, or a configuration object.
CLIENT: The entity that creates an INFERENCE REQUEST and consumes its result. A CLIENT can also operate other ODSI roles.
COORDINATION PLANE: The non-latency-critical functions used for discovery, MODEL MANIFEST publication, resource advertisement, path construction, evidence handling, governance, and optional settlement The term denotes functions and interfaces, not a required centralized controller.
DEADLINE: A locally interpreted time bound associated with a request or step. A DEADLINE is not globally enforceable evidence unless the selected profile defines a trusted time and observation model.
EXECUTION COMMITMENT: A signed statement binding a PARTICIPANT to a request, EXECUTION PROFILE, execution unit, input commitment, predecessor, and locally interpreted timing terms. A commitment attributes a statement; it does not prove correct computation.
EXECUTION PATH: An ordered set of execution assignments that, together, evaluate an INFERENCE REQUEST.
EXECUTION PROFILE: A versioned description of the semantics required to evaluate and verify a model, including partitioning, tensor formats, numerical behavior, decoding, and the VERIFICATION PROFILE.
EXECUTION RECEIPT: A signed record of an attempted execution unit, including input and output commitments, predecessor information, observed timing, and state information required by the selected verification method.
EXECUTION UNIT: The smallest computation independently assigned and receipted by an ODSI deployment. It can contain one layer, a group of layers, an expert, or another graph segment.
INFERENCE PLANE: The latency-sensitive functions that transport request data and state, execute assigned units, return receipts, and report failures.
INFERENCE REQUEST: A request bound to a MODEL MANIFEST, EXECUTION PROFILE, input, decoding parameters, and client policy.
MODEL MANIFEST: A signed, content-addressed description of the artifacts, graph, versions, compatibility information, and governance metadata that identify an executable model.
PARTICIPANT: An independently operated entity that contributes compute, memory, artifact storage, or network capacity using one or more cryptographic identities.
RESOURCE PROFILE: A signed, time-limited description of resources and supported execution capabilities advertised by a PARTICIPANT.
SETTLEMENT MECHANISM: An optional mechanism that records accepted contributions and transfers or accounts for value. Settlement is outside the inference critical path.
STATE HANDLE: A reference to per-request or reusable state, including its type, version, location or retrieval information, integrity commitment, and access policy.
VERIFICATION PROFILE: The selected method and parameters for deciding whether an execution result is acceptable and whether evidence is sufficient for a consequence.

System Model

Decentralization Properties ODSI separates five properties that are sometimes conflated as model ownership:

resource ownership: compute and storage are supplied by independent entities;
artifact custody: no single participant is required to retain every artifact;
operational control: no single participant is required to admit all nodes or schedule all work;
verification control: no single participant's assertion is sufficient in profiles intended to tolerate malicious execution; and
governance control: model and protocol changes follow the governance mechanism named by the deployment profile.

A deployment can decentralize these properties to different degrees. It needs to disclose which property remains centralized. ODSI does not label a system decentralized solely because it runs on geographically distributed nodes.

Roles The architecture defines the following logical roles. One participant can perform several roles, and a role can be replicated:

clients create requests and specify acceptable service, privacy, cost, and verification policy;
artifact providers publish or retain model artifacts;
execution nodes evaluate execution units and retain associated state;
discovery nodes help locate manifests, artifacts, resources, and peers;
path constructors select candidate execution assignments;
verifiers evaluate receipts or results under a verification profile;
evidence adjudicators apply profile-defined rules to objective evidence; and
settlement nodes record accepted contributions when settlement is enabled.

The last three roles are not automatically trustworthy. A deployment profile defines their selection, replication, independence assumptions, and decision rule.

Network and Timing Assumptions Participants communicate over an asynchronous packet network. Messages can be delayed, lost, duplicated, reordered, or replayed, and participants can become unreachable. ODSI does not assume a global upper bound on delay or globally synchronized clocks. Consequently, the architecture cannot guarantee both deterministic progress and safety under every partition or fault pattern . Interactive deployments can operate under periods of practical synchrony, using observed delay distributions and local timeout policies. A timeout is evidence that an observer did not receive a result in time; it is not, by itself, proof that a remote node failed to compute or transmit the result. Profiles that penalize deadline violations need an explicit trusted observation, timestamping, or multi-observer rule.

Adversary and Failure Model Participants can crash, disconnect, perform slowly, advertise false resources, return incorrect results, equivocate, replay old state, withhold artifacts, selectively abort, or collude. An adversary can create many identities, observe and modify traffic it controls, and attempt to isolate a participant from honest peers. The base architecture does not assume an honest global majority. A concrete profile that depends on an honest majority, honest stake fraction, trusted hardware operator, proof-system assumption, or independent verifier set needs to state that dependency and the consequence when it fails. "Majority" needs to name the measured resource, such as identities, stake, assigned executions, or verifier weight. Clients can also be malicious. They can submit resource-exhausting inputs, induce repeated path construction, dispute correct work, or attempt to extract model artifacts and other users' state.

Required Deployment Profile The architecture requies a versioned ODSI profile selects at least the following specifications before deployment:

identity, authentication, key rotation, and revocation mechanisms;
peer, manifest, artifact, and resource discovery mechanisms;
canonical encodings and digest algorithms for named objects;
internationalization, comparison, and display rules for human-readable fields, while keeping protocol identifiers unambiguous;
an EXECUTION PROFILE and compatible transport bindings;
a VERIFICATION PROFILE and its adversary assumptions;
a path-construction and admission policy;
failure, retry, state-recovery, and cancellation behavior;
privacy and transport-confidentiality behavior;
a governance mechanism for profile and model changes; and
Sybil resistance, evidence adjudication, and settlement behavior when those functions are enabled.

This list is a conformance boundary for future specifications. Implementations using different unresolved choices are not assumed to interoperate.

Named Objects and Execution Semantics

Model Manifest A MODEL MANIFEST provides stable identity for the computation requested by a client. At minimum, it contains or references:

a model identifier and manifest version;
the content digest, size, encoding, and retrieval information for every required artifact;
a graph that maps execution-unit identifiers to artifacts and their dependencies;
tokenizer, adapter, runtime, safety-component, and configuration identifiers;
supported EXECUTION PROFILE identifiers;
compatibility and deprecation information;
publisher signatures and the governance rule for authorized updates; and
licensing or usage-policy references where applicable.

Artifact digests detect corruption or substitution after a manifest is trusted. They do not establish that a model is safe, lawful, unbiased, useful, or free of malicious code. Trust in a manifest publisher and authorization of updates are profile-level decisions. An implementation can replicate or erasure-code artifacts, and no participant is required by this architecture to store a complete checkpoint. However, every execution unit needs timely access to its required artifacts. A profile needs an availability strategy that prevents a unique artifact holder from becoming a single point of failure.

Execution Profile An EXECUTION PROFILE removes ambiguity about what it means to execute a manifest. It defines:

execution-unit boundaries and their ordering;
tensor names, dimensions, element types, byte order, framing, and compression;
supported quantization and runtime variants;
state creation, lookup, migration, expiration, and invalidation semantics;
prefill, decode, batching, and cancellation behavior;
decoding parameters and the party responsible for stochastic sampling;
numerical equivalence rules; and
the identifier and parameters of the VERIFICATION PROFILE.

Floating-point results can differ across accelerators, kernels, batching choices, and compiler versions. A profile can require bitwise-deterministic execution, define a tolerance and comparison procedure, verify a deterministic representation such as quantized outputs, or use a proof or attestation mechanism. A digest comparison alone only works when the profile requires identical bytes. Stochastic sampling needs special treatment. A profile can bind a seed and algorithm, move sampling to the client, or verify logits before sampling. Without such a rule, two honest executions can produce different output tokens.

Request, Step, and State Identity An INFERENCE REQUEST is bound to the digest of its MODEL MANIFEST and EXECUTION PROFILE. It also identifies the input commitment, requested operation, decoding parameters, client policy, and a nonce that prevents accidental request collision. Every assigned step is identified by the request identifier, token or prefill position, execution-unit identifier, attempt number, and predecessor commitment. This tuple prevents a valid output for one position, model version, or retry from being substituted into another. A STATE HANDLE identifies the state type, producing step, manifest and profile versions, integrity commitment, access scope, expiration policy, and available locations. Possession of a state handle does not itself authorize access to the state.

Resource Profile A RESOURCE PROFILE describes what a participant offers rather than exposing every hardware detail. It can include:

supported execution units, profiles, runtimes, and artifact residency;
accelerator and memory classes;
capacity, queue, throughput, and observed execution-time ranges;
network reachability and locality information;
state capacity and migration support;
verification or attestation capabilities;
administrative, jurisdictional, and privacy attributes; and
validity interval, sequence number, and participant signature.

Fast-changing values are claims, not facts. Path construction needs to account for measurement source, freshness, uncertainty, and the cost of verification. Resource profiles use application-level information; raw values are not automatically suitable for distribution as CATS metrics.

Commitments, Receipts, and Evidence Before accepting an execution assignment, a node can issue an EXECUTION COMMITMENT binding its identity to the request, execution unit, input commitment, predecessor, execution profile, attempt, and agreed service terms. After execution, the node produces an EXECUTION RECEIPT containing the corresponding identifiers, output commitment or output, resulting state commitment, observed local timing, and verification-specific material. Receipts are linked to predecessor receipts so that omission, replay, and equivocation can be investigated. Signatures provide origin authentication and integrity for these statements. They do not prove that advertised resources existed, that computation occurred, that an output is correct, or that a deadline was objectively missed. Those conclusions require evidence defined by the selected profile.

ODSI Architecture

Functional Overview The architecture separates latency-sensitive inference from slower coordination and optional settlement: Exec B ---> Exec C ---> ... | | unit 0 units 1-3 unit 4 | | activations, state handles, receipts | +------------------------+------------------------+ | events and observations | +------------------------v------------------------+ | Coordination Plane | | manifest/artifact discovery | resource discovery| | path construction | verification | | evidence adjudication | governance | +------------------------+------------------------+ | accepted contribution | +------------------------v------------------------+ | Optional Settlement Mechanism | +-------------------------------------------------+ ]]> The diagram shows functional relationships, not central services. Each coordination function can be distributed or replicated according to the deployment profile.

Protocol-Independent Interfaces Future ODSI protocol specifications need to cover the following interfaces:

manifest and artifact publication, retrieval, validation, and withdrawal;
peer and resource discovery, including freshness and anti-replay behavior;
execution solicitation, offer, selection, and commitment;
activation and state transport with request and predecessor binding;
execution receipt and failure reporting;
verification request, evidence exchange, and decision reporting;
cancellation, retry, replacement, and state recovery; and
contribution submission to an optional settlement mechanism.

These interfaces can use different protocols. A deployment profile identifies the binding between them and prevents downgrade to weaker identity, privacy, or verification behavior.

Path Construction A path constructor selects an ordered set of assignments compatible with the manifest, execution profile, resource profiles, client policy, and current observations. Path construction can be performed by a client, a set of brokers, an auction, a distributed algorithm, or another profile-defined mechanism. ODSI does not require a globally optimal path or a single scheduler. A candidate path needs to account for compute time, queueing, artifact availability, activation transfer, state affinity, verification overhead, failure probability, diversity, privacy policy, and cost. Selecting each unit independently can create a path whose slowest boundary dominates the token rate. Implementations therefore commonly assign contiguous units and favor stable paths across token steps. If the verification profile relies on redundant execution, path construction also enforces its independence or diversity assumptions. Merely choosing two identities is insufficient when both can be controlled by one entity.

Protocol-Independent Workflow

Participant Join and Resource Publication

First, a participant creates or obtains a profile-compatible identity and discovers bootstrap peers through the selected discovery mechanism.
The participant then obtains the active ODSI profile and the manifests it intends to support. It validates their governance and signatures before retrieving artifacts.
The participant validates artifact digests, prepares compatible runtimes, and publishes a signed, expiring RESOURCE PROFILE.
Finally, discovery nodes distribute or index the profile according to the profile's privacy, freshness, and Sybil-resistance rules.

Joining discovery does not automatically grant assignments. Path constructors apply admission, diversity, reputation, collateral, attestation, or challenge policies defined by the deployment profile.

Request and Path Establishment

First, a client selects a MODEL MANIFEST, EXECUTION PROFILE, and client policy. The policy includes acceptable verification, confidentiality, latency, cost, and jurisdiction constraints.
A path constructor then obtains candidate resource profiles and recent observations. It builds one or more feasible paths and reserves resources if the selected protocol supports reservation.
Selected execution nodes validate the request and predecessor information, then return commitments or rejection reasons.
Finally, the client or path constructor activates a path only after the commitments required by its verification and failure policy are available.

Inference Execution

First, the initial assigned node receives a request-bound input and verifies its manifest, profile, step, attempt, and predecessor fields.
It then evaluates its assigned execution unit, retains or emits state as specified, and sends the next activation together with a receipt or receipt reference.
Each successor rejects inputs that are stale, duplicated, incompatible, unauthorized, or not linked to the expected predecessor.
Finally, the final node returns the profile-defined result and receipt chain to the client. Verification work can occur inline, asynchronously, or both.

An asynchronous verifier cannot retract an output already acted upon by a client. Applications that require prevention rather than later detection need inline verification or a trusted execution path and must include its latency in the service objective.

Failure and Path Repair When a local timeout or explicit failure occurs, the path constructor can retry the same node, use a warm replica, reassign downstream work, migrate state, reconstruct state from an earlier checkpoint, or restart the request. The selected action depends on remaining client time, state size, artifact availability, and verification policy. Every retry receives a new attempt identifier. Late results from an earlier attempt cannot be inserted into the repaired path. A repaired path records the last accepted predecessor so verifiers can distinguish a valid retry from equivocation. Path repair is not always transparent. Loss of unique KV state can require replay of all preceding tokens, and loss of a unique model artifact can make progress impossible. Profiles intended for interactive service need replication or checkpoint rules that bound these risks under their stated failure assumptions.

Verification and Settlement Receipts and verification material are evaluated according to the VERIFICATION PROFILE. An accepted verification decision identifies the evidence, decision rule, participating verifiers, and confidence or finality semantics. Only accepted contributions are submitted for settlement. A settlement mechanism can aggregate per-step receipts rather than place every inference event in a global consensus log. Disputed contributions remain pending until the profile-defined adjudication rule resolves or expires them.

Correctness and Verification

Correctness Definition For ODSI, an execution result is correct only relative to a specific manifest and execution profile. Correctness includes:

the expected execution unit consumed the expected predecessor output;
required artifacts and state versions were used;
the output satisfies the profile's numerical-equivalence rule;
execution did not violate a profile-defined policy that is part of correctness; and
the result belongs to the identified request, position, and attempt.

Availability and deadline adherence are service properties, not computational correctness. A correct late result and a timely incorrect result need different handling.

Verification Methods No single verification method is suitable for all models and deployments. A VERIFICATION PROFILE can use one or more of the following:

redundant execution compares independently produced results; it adds compute cost and requires a decision rule for numerical differences and collusion;
challenge-based recomputation samples steps or subcomputations; it reduces average verification cost but detects cheating probabilistically and needs unpredictable challenges;
trusted execution evidence can bind code and artifacts to a hardware-backed environment; it introduces hardware, manufacturer, freshness, side-channel, and revocation assumptions;
cryptographic validity proofs can provide strong verification for supported computations; they introduce prover cost, circuit or model-binding complexity, setup assumptions, and potentially substantial latency; and
client or verifier recomputation provides a direct reference under deterministic semantics but can eliminate the resource advantage that motivated outsourcing.

A profile documents which faults its method detects, its false-accept and false-reject behavior, independence assumptions, and whether a decision is immediate, probabilistic, or eventual.

Evidence and Consequences Objective evidence can include conflicting signed commitments, two incompatible receipts for the same step and attempt, an invalid artifact digest, or verification material that fails a deterministic check. Signed timing assertions from one party are not objective proof of remote lateness. Consequences need to be proportional to evidence. A profile can distinguish:

service failure, which affects selection or payment but is not proven misconduct;
attributable protocol violation, such as equivocation or replay;
verified incorrect execution under the selected verification method; and
inconclusive disagreement, which can trigger further verification without punishment.

This distinction prevents network delay, verifier disagreement, or honest numerical variation from automatically causing a penalty.

Timing, State, and Recovery

Deadline Semantics The client defines an end-to-end service objective, such as time to first token or time per output token. A path constructor can allocate local budgets to execution and transfer steps. These budgets guide scheduling and recovery but do not create globally comparable timestamps. Remaining slack is an estimate derived from client-observed time, predicted compute and transfer time, and verification overhead. Nodes can report local durations, while receivers can measure arrival times. A profile that needs cross-node timing evidence specifies clock synchronization, error bounds, trusted timestamp sources, and observation rules. ODSI cannot guarantee an upper latency bound over an asynchronous public network. Implementations can report percentile objectives and failure probabilities for a measured operating region.

Stateful Execution KV caches and other request state can be much larger than an individual activation. Stable assignment across decode steps avoids repeated state movement, while reassignment can require migration or replay. A state policy defines:

which node owns the authoritative current version;
whether and when replicas or checkpoints are created;
how state is encrypted and authorized;
how concurrent attempts are fenced;
how rollback and stale-state use are detected; and
when state is deleted.

State reconstruction needs the exact manifest, profile, request prefix, and accepted receipt history. If these are unavailable, restarting from a claimed state can compromise correctness.

Churn and Availability Open membership makes churn normal. Availability depends on replication of artifacts, diversity of compatible execution nodes, repair capacity, and the amount of non-replicated request state. Reputation or economic penalties do not restore unavailable data and cannot substitute for redundancy. Profiles need explicit minimum replication or recoverability policies for critical artifacts and state. They also need admission control so that apparent resource growth does not overload discovery, verification, or path-construction functions.

Incentives, Sybil Resistance, and Governance

Incentive Scope Independent resource owners incur costs and may require compensation. ODSI therefore supports contribution accounting, but it does not require a particular currency or assume that payment alone causes honest behavior. An accounting rule identifies the unit being purchased, the evidence required for acceptance, cancellation behavior, verification cost, and who pays for retries. Paying solely for submitted outputs encourages spam; paying solely for accepted outputs can expose honest nodes to malicious clients or faulty verifiers. These trade-offs are profile and deployment policy.

Sybil and Collusion Resistance Cryptographic identities make actions linkable but do not make entities unique. A profile can use collateral, proof of resource, rate limits, admission attestations, diversity constraints, historical performance, or combinations of these mechanisms. Each choice changes openness and concentration risk. Stake makes identities costly only relative to the value and liquidity of the stake and the credibility of enforcement. It does not prove resource ownership, prevent collusion, or establish correct execution. Reputation can also be manipulated through self-dealing, selective behavior, identity retirement, and reputation farming.

Settlement and Adjudication If a deployment uses a ledger or replicated state machine, its consensus and finality assumptions are part of the ODSI profile. Byzantine consensus techniques illustrate how replicated parties can agree under bounded adversary assumptions , but ODSI does not assume that one such technique is universally suitable. Penalties are applied only by an authorized settlement or governance mechanism and only for profile-defined evidence. Missed local deadlines should normally affect payment or future selection rather than trigger punitive slashing unless the profile provides objective timing evidence.

Model and Protocol Governance Decentralized resource contribution does not determine who can publish a model version, change safety policy, update a verification method, or repair a vulnerability. A deployment profile names the governance rule for:

manifest publication and revocation;
protocol and profile upgrades;
emergency suspension;
verifier and adjudicator changes; and
incompatible forks.

Clients need to bind requests to explicit versions rather than an ambiguous "latest" model. Participants can support several governance domains without treating them as one inference service.

CATS Applicability ODSI and CATS operate at different scopes but can be composed. Within a participant's provider-controlled network, a CATS Path Selector can select among service contact instances that provide an ODSI execution capability. Network and compute conditions can help select the local instance. The result is then exposed to the ODSI overlay through one participant identity and resource profile. The following conceptual mapping can be useful inside such a domain:

an ODSI execution endpoint can be realized as a CATS service instance;
a compatible execution-unit capability can contribute to identification of the offered service;
CATS network and compute information can inform the participant's local endpoint selection; and
the participant advertises a service-oriented summary to ODSI rather than exporting raw CATS state across domains.

This composition preserves the CATS single-provider boundary. ODSI does not require a CATS Path Selector to understand global stake, reputation, model governance, or receipt adjudication. It also does not require CATS to steer every layer boundary or expose rapidly changing application state to the network. Before defining ODSI-specific CATS metrics, deployment experience is needed to determine which information is stable, comparable, safe to disclose, and useful for traffic steering. Model identifiers, tensor details, and settlement attributes are application metadata and are not presumed to be CATS metrics.

Scalability and Performance Considerations For an execution path with n units, token-step latency includes the sum of unit execution time, queueing time, inter-unit transfer time, state operations, and inline verification. Throughput is constrained by the slowest repeatedly used stage and by pipeline utilization. Adding participants increases the candidate resource pool but does not necessarily reduce latency or increase useful capacity. Important scaling limits include:

activation size and the number of wide-area stage boundaries;
model-artifact loading and warm-up time;
KV-cache growth and recovery cost;
stragglers and correlated failures;
discovery and resource-advertisement churn;
verification and redundant-execution overhead;
settlement event volume; and
concentration around nodes with rare artifacts, high bandwidth, or strong reputation.

Every transport binding needs congestion control appropriate to the selected transport. Deadline pressure does not justify unbounded retransmission, duplicate execution traffic, or activation bursts. Path repair and redundant verification need to share congestion and backpressure information with ordinary inference traffic so that recovery from one failure does not cause a wider overload. Research systems have demonstrated collaborative inference across multiple parties and adaptive pipelines over heterogeneous unreliable devices . These results motivate the architecture but do not establish performance for public-Internet deployment or adversary model. An ODSI evaluation should report model and execution profiles, participant count and diversity, network distributions, activation and state volume, time to first token, time per output token, throughput, tail latency, repair time, verification cost, false decisions, and behavior under churn and adversarial actions. Comparisons should include a provider-controlled distributed baseline. Claims of scalability, bounded recovery, or cost reduction need to identify the measured operating region.

Manageability and Operational Considerations Even without one global operator, each participant needs local management and the overlay needs interoperable observations. Implementations should expose privacy-preserving counters and failure reasons for:

manifest, artifact, and profile validation;
peer discovery and resource-profile freshness;
assignment acceptance, rejection, cancellation, and retry;
compute, queue, transfer, state, and verification duration;
state creation, replication, migration, expiration, and loss;
receipt-chain gaps, equivocation, and verification outcomes; and
settlement submission and finality state when enabled.

Failure reports need stable categories so that a path constructor can distinguish incompatibility, overload, policy rejection, missing artifact, missing state, transport failure, invalid predecessor, and verification failure. Detailed diagnostics can reveal topology, capacity, model holdings, or user behavior and therefore require access control and retention limits. Participants need procedures for key compromise, vulnerable artifacts, verifier failure, profile rollback, and emergency manifest revocation. Because coordination is decentralized, revocation propagation can be delayed or partitioned. Clients should treat the freshness of governance and revocation state as part of path acceptance.

Security Considerations This architecture assumes hostile participants and networks. A concrete protocol specification is expected to follow the threat-analysis guidance in and to state how it addresses the threats below.

Identity and Message Security Self-generated keys provide stable pseudonyms, not real-world identity, uniqueness, authorization, or trust. Protocol bindings need mutual authentication where required, integrity protection, replay protection, context binding, algorithm agility, key rotation, and revocation. Every signed object needs a domain separator and version so that a valid signature in one context cannot authorize another action. Traffic between execution nodes can contain prompts, activations, state, results, and receipts. It requires confidentiality and integrity protection against on-path attackers unless a deployment explicitly accepts exposure. Transport security does not protect data from an execution node that legitimately receives plaintext.

Manifest and Software Supply Chain A malicious or compromised manifest publisher can distribute backdoored weights, unsafe runtime code, incompatible tokenizers, or weakened verification settings. Artifact digests only preserve the published bytes. Profiles need publisher authorization, review or transparency policy, rollback protection, revocation, dependency validation, and isolation of untrusted runtime components. Content-addressed retrieval can expose which models a participant supports. An attacker can withhold rare artifacts, advertise unavailable pieces, or flood storage with low-value artifacts. Replication policy, quotas, proof of possession where appropriate, and retrieval diversity reduce these risks.

Resource Discovery and Path Manipulation Attackers can create Sybil identities, advertise false performance, replay fresh-looking profiles, eclipse clients, bias peer sampling, or attract sensitive requests before failing or observing them. Signed advertisements do not establish truth. Profiles need freshness limits, rate controls, independent observations, peer diversity, and resistance to a single discovery source. Path constructors can censor participants, leak request policy, select colluding verifiers, or create paths that concentrate information. Clients should be able to validate path constraints and, where the threat model requires it, obtain paths from independent sources or construct paths locally.

Execution Integrity An execution node can use wrong artifacts or state, skip computation, fabricate output, return stale output, equivocate, or selectively corrupt requests. Commitments and receipts support attribution but do not prove computation. The VERIFICATION PROFILE is the primary mitigation and needs to address numerical nondeterminism, verifier collusion, adaptive cheating, and false accusation. Redundant execution fails when replicas collude or share a common faulty implementation. Trusted execution can fail through compromised hardware, stale attestation, side channels, or vendor control. Proof systems can contain implementation errors or bind to the wrong model. The profile needs operational response for failure of its verification root.

State and Replay Attacks State can be rolled back, substituted between users, replayed across attempts, or retained after expiration. Request-, manifest-, profile-, position-, and predecessor-binding prevent many substitutions. Concurrent retries need fencing so that two valid branches do not both become authoritative. State needs access control, integrity protection, lifecycle enforcement, and secure deletion appropriate to the deployment.

Denial of Service Clients can submit long contexts, request unsupported profiles, repeatedly trigger repair, or create expensive disputes. Participants can accept work and withhold results, overload successors with large activations, or flood discovery and verification systems. Implementations need admission control, authenticated quotas, bounded parsing and allocation, request-size limits, cancellation, backpressure, and separation between untrusted input processing and scarce accelerator resources. Settlement can amplify denial of service if every event requires global consensus. Aggregation, channels, batching, and minimum-value policies can reduce event volume, but their trust and failure behavior must be documented.

Economic and Governance Attacks Stake and reputation introduce bribery, collusion, self-dealing, identity farming, verifier capture, censorship, and concentration risks. A party can behave honestly while building reputation and cheat only on selected valuable requests. Governance participants can authorize a malicious model or verification downgrade. Economic parameters and governance are therefore security-critical configuration, not merely business policy. Penalizing ambiguous failures creates an attack in which an adversary delays traffic or supplies inconsistent observations to cause loss to an honest node. Punitive consequences require stronger evidence than non-payment or path-selection decisions.

Residual Risk No base ODSI mechanism guarantees correct, confidential, available, or timely inference. Those properties depend on the selected deployment profile and its assumptions. Applications with safety-critical consequences should not act on optimistic results before the verification required by their risk model completes.

Privacy Considerations ODSI increases the number of administrative entities that can observe parts of an inference request. Prompts, tokens, activations, logits, embeddings, KV caches, model-selection information, timing, path structure, payment records, and resource advertisements can reveal user content or behavior. Splitting a model does not provide a general confidentiality guarantee. Intermediate representations can retain information about inputs, and a participant can combine observations across requests. Likewise, distributing execution closer to a user does not improve privacy if more independent parties receive the data. A deployment profile needs to specify:

which roles receive plaintext input, activations, state, and output;
end-to-end and hop-by-hop encryption boundaries;
participant-selection and jurisdiction constraints;
unlinkability or pseudonym-rotation behavior and its interaction with reputation;
logging, evidence, and settlement retention;
state isolation and deletion; and
whether verification reveals inputs or intermediate values.

Data minimization can favor larger contiguous execution units so fewer parties observe intermediate values. Trusted execution, secure multiparty computation, homomorphic techniques, or zero-knowledge proofs can change exposure, but each adds assumptions and cost and is not provided by this architecture. Public settlement records can permanently link participants, clients, timing, and workload volume. Profiles should avoid placing request content or stable user identifiers in globally replicated records and should disclose unavoidable metadata leakage.

IANA Considerations This document has no IANA actions.

Relationship to Existing Work IDN distributes and selects inference capabilities within one provider-controlled domain. ODSI uses a different trust model and complements rather than replaces IDN. An IDN deployment can contribute an endpoint to ODSI, while remaining internally provider controlled. CATS combines network and compute information when steering traffic to service instances. ODSI can use CATS inside a participant domain but keeps cross-domain peer and execution coordination at the application layer. Peer-to-peer systems demonstrate content-addressed distribution and incentive mechanisms . Decentralized ledgers demonstrate replicated settlement under explicit consensus assumptions . ODSI borrows architectural lessons but does not require either protocol. Collaborative model-serving systems demonstrate that multiple parties can collectively host large models . Work on unreliable heterogeneous pipelines explores adaptive placement under churn . ODSI focuses on the Internet architecture, trust boundaries, interoperable objects, and evidence requirements that arise when such participants do not share one operator.

Acknowledgments The authors would like to thank colleagues and reviewers in the community who provided feedback on the early version of this draft.