Protocol Conformance Test Evidence Records: A Reproducibility Profile

Introduction Software protocol implementations are evaluated by test harnesses, continuous-integration jobs, conformance suites, and interoperability labs. Reports can be difficult to compare when they omit artifact digests, test-suite versions, runtime environments, case-level status, or provenance linking results to source inputs. This document defines a small, extensible JSON record profile for protocol-conformance test evidence. A record states what its producer asserts happened during a run. It is not proof that the harness was trustworthy, the test suite complete, or the implementation secure. Existing work covers substantial adjacent requirements. SCITT defines transparency for signed statements; RATS defines evidence-appraisal roles and terminology; COSE and JSON Canonicalization Scheme provide existing security/canonicalization building blocks; and in-toto supports authenticated metadata with type-specific predicates. Implementers SHOULD first evaluate whether existing formats meet their needs.

Conventions and Terminology The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL are to be interpreted as described in BCP 14 (RFC 2119 and RFC 8174) when, and only when, they appear in all capitals.

Evidence record: A structured statement describing a test run and its reported outcomes.
Subject artifact: The implementation, binary, source snapshot, endpoint, or object evaluated by the test run.
Test suite: The named and versioned collection of cases executed by a harness.
Producer: The system that creates the record.
Consumer: A system or person that parses and evaluates the record.

Scope and Non-Goals The profile describes test-run inputs, environment, case-level outcomes, artifact digests, provenance, and limitations. This document does not specify test design, pass/fail computation, key provisioning, signing, transparency receipts, or trust decisions. Those functions SHOULD use established mechanisms. This document does not establish a new evidence ledger. Deployments requiring transparency SHOULD evaluate SCITT (RFC 9943); deployments requiring remote-attestation semantics SHOULD evaluate RATS (RFC 9334) where applicable.

Evidence Record Model A record is a UTF-8 JSON object. Producers MUST include schema_version, record_id, subject, test_suite, implementation, environment, run, cases, result_summary, and provenance. Consumers SHOULD preserve unknown extension members when transforming or re-emitting records.

Subject and Test SuiteThe subject SHOULD identify the artifact by stable name and cryptographic digest. The test_suite SHOULD identify suite name, version, and digest. Digests MUST identify exact bytes or a clearly specified canonical representation; a label alone is not a stable artifact identifier.

Environment and RunThe environment SHOULD identify relevant runtime, operating system, architecture, dependencies, and hardware. The run MUST include status and SHOULD include timestamps, exit code, case count, and assertion count when available. Missing values MUST NOT be fabricated.

Case-Level OutcomesEach case MUST have a stable case_id and status of pass, fail, skip, error, or not-run. Expected and observed values MAY be included. Logs SHOULD be referenced by digest or controlled URI rather than embedded when they may contain secrets or personal data.

Provenance and LimitationsThe provenance member SHOULD link source revisions, test-harness digests, build identifiers, and producer identity. A limitations array is RECOMMENDED and SHOULD describe untested platforms, incomplete coverage, simulated inputs, and other material constraints.

Canonicalization and Integrity JSON does not inherently provide a unique byte representation for semantically equivalent objects. If JSON is signed or hashed, implementations SHOULD use an established canonicalization procedure such as RFC 8785 and MUST document which bytes are covered. This profile defines no signature format. Signed statements SHOULD reuse an established mechanism such as COSE (RFC 9052) or an existing attestation envelope. Where transparency is required, deployments SHOULD evaluate SCITT (RFC 9943). A signature authenticates an assertion under the selected trust model; it does not prove the assertion true.

ExampleThe following example is illustrative, unsigned, and has no transparency receipt. "}, "test_suite": {"name": "example-suite", "version": "1.0", "digest_algorithm": "sha-256", "digest": ""}, "implementation": {"name": "example-implementation", "version": "1.0"}, "environment": {"runtime": "example-runtime", "platform": "example-platform"}, "run": {"status": "passed", "case_count": 2, "assertion_count": 4, "exit_code": 0}, "cases": [{"case_id": "CASE-001", "status": "pass"}, {"case_id": "CASE-002", "status": "pass"}], "result_summary": {"passed": 2, "failed": 0, "skipped": 0}, "provenance": {"producer": "example-harness", "source_revision": ""}, "limitations": ["Illustrative example; not a measurement"] } ]]>

Interoperability Considerations Producers and consumers SHOULD test this profile against existing test-report formats and attestation predicates before adoption. Consumers MUST distinguish an unknown extension from a verified security property. Conformance vectors SHOULD cover pass, fail, skip, error, not-run, missing digest, and incomplete provenance. Before standardization, compare the proposed field set with in-toto predicates, TAP, JUnit-style reports, and SCITT statement payloads. If an existing format expresses the required information without ambiguity, reusing it is preferred.

Security Considerations An evidence record is an assertion by its producer. A compromised or dishonest producer can report false results, omit failures, or associate results with the wrong artifact. Consumers SHOULD verify digests independently, authenticate the producer where needed, and apply local trust policy. Records can be replayed or presented out of context. Deployments SHOULD use unique run identifiers, trustworthy timestamps where available, immutable source revisions, and transparency/attestation mechanisms where justified. A digest alone does not establish freshness, authorship, correctness, or completeness. Logs and environment metadata can reveal secrets, hostnames, tokens, identifiers, vulnerabilities, or confidential details. Producers MUST NOT include secrets in public records and SHOULD minimize sensitive fields. This profile does not guarantee post-quantum security.

Privacy ConsiderationsEnvironment metadata may include personal or indirectly identifying information. Producers SHOULD collect only what is necessary, redact direct identifiers, and review log references before publication. Hashing a low-entropy identifier does not necessarily anonymize it.

IANA ConsiderationsThis document requests no IANA actions.

Status and Standards-Track CaveatThis individual draft candidate has not been submitted to the IETF, has no assigned Datatracker identifier, and represents no IETF consensus or endorsement. The prior-art review is preliminary. SCITT (RFC 9943), RATS (RFC 9334), in-toto attestations, and existing test-report formats overlap with parts of this proposal. Community feedback and an interoperability prototype are prerequisites for deciding whether a separate profile is useful.