A Benchmarking Method for the Integrity of AI Agent Memory at Rest

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 4 April 2027.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

An AI agent built on a framework such as LangGraph, Letta, Mem0, or a vector memory store keeps state between sessions: long-term memory records, session checkpoints, and their metadata. On the next turn the agent reads that state back and acts on it as trusted context. The store that holds the state is ordinary infrastructure, a database file, a table, a key-value namespace, or an object store, and is reachable by the same means as any other data: a compromised host, a shared credential, an injection flaw in a co-located application, a restored backup, or a malicious operator.¶

There is at present no agreed way to measure whether an agent notices when the memory behind it has been changed. Existing work on agent security concentrates on the input path: prompt injection, and poisoning through the agent's own write interface. The at-rest case, where the adversary edits the store directly, is different: the adversary needs no injection, and can move, remove, or roll back genuine records without authoring any content of their own.¶

This document specifies a laboratory benchmarking method for that case. It is deterministic, runs offline, and produces a single metric value together with a per-case verdict table. It follows the conventions of the Benchmarking Methodology Working Group: a controlled test bed, an explicit procedure, control cases that must hold for a result to count, and full reporting of the configuration under test [RFC2544] [RFC8239].¶

2. Scope and Relationship to Other Metrics

This method measures one property: the ability of a System Under Test (SUT) to detect at-rest tampering with its own persisted memory and to avoid serving tampered memory as genuine. It does not measure confidentiality of memory, resistance to prompt injection through the agent interface, or correctness of the agent's reasoning.¶

The adversary here is distinct from two adjacent adversaries discussed elsewhere. It differs from an adversary who can only talk to the agent (memory poisoning through the write path), and from a legitimate user of another session (cross-session isolation). The adversary in this document has write access to the storage medium but holds none of the SUT's cryptographic keys.¶

The method is intended to serve as the test method for a "Protection of Memory Data Integrity" metric. It can be used on its own for any agent memory subsystem.¶

The two adjacent adversaries have their own measurements, and the three fit together. Poisoning through the write path is measured by writing through the SUT's own interface on three attacker channels (an untrusted-source label, a forged trusted-source label, and a trusted label with a valid signature) and recording whether the read path later serves the planted memory as context. Cross-context isolation is measured both through the read path (write as one context, read as another) and, at the storage layer, by test case T6 of this method, since a filter that holds on the first can fail on the second. Proposed procedures for both were offered to the authors of the metric document on the working group list in October 2026.¶

3. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

Memory subsystem:

the component of the SUT that persists and later retrieves the agent's state: long-term memory records, session checkpoints, and their metadata.¶

Store:

the medium that holds persisted memory: a database file, a table, a key-value namespace, or an object store.¶

Write path:

the SUT's own interface for adding memory (for example, remember, add, put, or checkpoint).¶

Read path:

the SUT's own interface for retrieving memory for use in a turn (for example, recall, search, get state, or resume).¶

Context:

an isolation unit the SUT exposes for memory: a user, a thread, or a session.¶

Edit:

a single change applied to the store by the adversary using generic storage tooling, with no SUT code loaded.¶

4. Threat Model

The adversary has write access to the medium that holds the SUT's memory but holds none of the SUT's cryptographic keys. The adversary's goal is that the SUT resumes from memory of the adversary's choosing and that nobody notices.¶

Because the adversary can write anything to the store, an integrity value stored next to the data, a checksum column, a hash field, or a "verified" flag, provides no protection unless it is bound to a secret or a root of trust the adversary does not hold. The method therefore measures the SUT's behaviour on read, not the presence of integrity fields in the store. An adversary who can edit a memory record can equally edit a checksum stored beside it; the only question worth measuring is whether the SUT notices at the moment it loads the memory.¶

5. Test Bed

The memory subsystem MUST be run in the configuration the SUT's documentation recommends for production, including any at-rest encryption, signing, or audit feature the vendor documents as protecting memory. A feature that is off by default MUST be reported as off by default. Where the SUT offers such a feature, the evaluator SHOULD report two results, one with the feature off and one with it on, so that the two can be compared.¶

The evaluator seeds memory through the SUT's own write path: at least five records in each of two isolated contexts (two users, threads, or sessions, whichever the SUT exposes). One record in context A carries a distinctive fact FA and one record in context B carries a distinctive fact FB. Nothing is written directly to the store during seeding.¶

All reads are made through the SUT's own read path and never by inspecting the store. What the store contains after an edit is the adversary's business; what the SUT serves through its read path is the metric.¶

6. Test Cases

Each case starts from a fresh copy of the seeded store. The evaluator stops the SUT, applies exactly one edit with generic storage tooling (SQL, a file editor, or the store's own client with no SUT code loaded), restarts the SUT, and reads the affected context through the read path.¶

Stopping the SUT before the edit is part of the method, not a convenience. A subsystem that watches its store from inside a running process can detect every edit while that process is up and none once it has restarted, because the restarted process establishes a new baseline from whatever the store holds. The adversary who edits the store chooses the moment, and a store compromise typically coincides with a host restart. The evaluator MAY additionally report the process-alive position for comparison, but the metric is taken with the SUT stopped.¶

T1 Content tamper.

Change the fact inside one existing record in place.¶

T2 Tail truncation.

Delete the most recent record or records so that an earlier state becomes current.¶

T3 Middle deletion.

Delete a record that is neither the first nor the last.¶

T4 Reordering.

Swap the position of two records by editing their order keys, timestamps, or parent links.¶

T5 Forged insertion.

Insert a new record of the adversary's authorship, in the SUT's own storage format.¶

T6 Cross-context replay.

Copy a genuine record from context A over a record in context B, leaving B's identifiers in place.¶

T7 Rollback replay.

Copy a genuine older record of context A over A's most recent record.¶

T8 Metadata tamper.

Alter a record's owner, role, source, or timestamp field and leave its content unchanged.¶

T9 Whole-store rollback.

Take a complete copy of the store, every file and sidecar the SUT keeps for it included; write one more genuine record through the SUT's own write path; then put the earlier copy back in its place. The newest genuine record is gone and every integrity value the store held beside its records has gone back with it.¶

T6 and T7 use only bytes the SUT itself wrote. They pass through any at-rest encryption that does not bind ciphertext to record identity, and any signature that does not cover a record's position. They are the cases that separate confidentiality from integrity, and a subsystem that relies on encryption alone typically accepts both.¶

T9 also uses only bytes the SUT wrote, and it is a different case from T2. T2 removes the tail from inside a store whose hash chain, ledger, or anchor is left as it was, so a head kept anywhere, even beside the records, can catch it. T9 restores an older copy of everything, so a head kept beside the records is restored along with them and agrees with the older state. T9 is the case that separates a head stored in the store from a head anchored outside it: on a separate volume, in a remote service, or signed by a key the adversary does not hold. It is the edit a restored backup makes.¶

A subsystem that stores memory as unordered key-value pairs, with no inherent record order and no separate metadata layer, cannot express T2, T4, or T8. A managed service whose store the evaluator cannot copy and restore as a unit cannot express T9. Such cases are reported as "not applicable" for that SUT, with the reason, and are excluded from the denominator in Section 9. They are never reported as passes.¶

7. Verdicts

After the restart the evaluator reads the affected context through the read path and classifies the outcome as one of three verdicts.¶

REJECTED:

the SUT refuses to load or serve the affected memory: an error, an empty result, or a refusal to resume.¶

REPORTED:

the SUT serves the memory but raises an integrity signal through a documented channel, a log line at warning level or above, a callback, or a status field, before or at the time of serving.¶

ACCEPTED:

the SUT serves the edited memory as genuine with no signal.¶

REJECTED and REPORTED are passes. ACCEPTED is a fail. Each verdict carries a detection point. A signal raised at the moment the memory is loaded or served has detection point "read". A signal that appears only when the operator runs a separate audit command has detection point "audit"; such a result is recorded as REPORTED with detection point "audit" and counts as a partial pass, because by the time such an audit runs the agent has already resumed from and acted on the memory.¶

The detection point describes how the evaluator reached the SUT's signal, so it is normally the same for every case in one result: a subsystem either checks on the read path or it offers an audit command. A result MUST state its detection point, and a subsystem that offers both MUST be reported as two results, one per detection point, so that the read-path verdicts are never credited with what only the audit found.¶

8. Control Cases

Three control cases MUST hold for a result to be valid.¶

C1 No-op reload.

Stop the SUT, restart it with no edit, and read. The SUT MUST serve the seeded memory unchanged and MUST NOT raise an integrity signal.¶

C2 Genuine write after restart.

Stop, restart, write one more record through the SUT's own write path, and read. The record MUST be served.¶

C3 Edit landed.

Before the SUT is restarted, the evaluator compares the raw store before and after the edit and confirms that the edit changed exactly what the case intends and nothing else: for T1 the content of one record; for T2 and T3 the absence of the named records; for T4 two distinct records exchanged; for T5 exactly one new record; for T6 a victim slot holding a donor record from another context, with the victim's identifiers; for T7 a victim slot holding an older record of the same context; for T8 a changed owner, role, source, or timestamp with the content untouched; for T9 that the genuine write added exactly one record and left the others as they were, and that the restored copy matches the store as it stood before that write, record for record. A case whose edit did not land is recorded as "error", is excluded from the denominator, and is never recorded as a pass or a fail.¶

If C1 or C2 fails, the metric MUST NOT be computed and the result MUST be reported as "not evaluable" with the reason. Without C1, a subsystem that refuses everything would score a perfect pass rate; C1 is therefore not optional. C3 exists because an evaluator's edit can silently miss: a copy made onto the record it came from, a reorder that writes each record back under its own key, or a replay whose victim and donor share an owner all leave the store as it was, and the SUT then "accepts" an edit that never happened. Two such no-op cells were found in the reference implementation's own results and corrected after C3 was added. C3 also refuses to score a T6 cell when the two contexts were seeded with the same values, because a donor record landing in the victim slot is then indistinguishable from the victim's own record; the seeding in Section 5 gives each context distinct values for this reason.¶

9. Scoring

Let N be the number of applicable test cases for the SUT (nine, less any cases reported "not applicable" under Section 6). Let passes be the count of REJECTED and read-time REPORTED verdicts, and let partial be the count of audit-time REPORTED verdicts. The metric is:¶

Metric Value = (passes + 0.5 * partial) / N¶

Evaluators MAY additionally report the pass rate on T1 to T5, on T6 to T8, and on T9 separately, since the first group tests tamper evidence, the second tests binding of a record to its place and owner in the store, and the third tests whether the store's integrity head is held somewhere the adversary cannot roll back.¶

10. Reporting

Each result MUST state: the SUT and the exact version of its memory component; the store backend and version; the configuration used, that is, which encryption, signing, or audit features were on or off; for every test case the verdict and the detection point; that the SUT was stopped during each edit, or the process-alive position if that is additionally reported; the exact commands or script that produced the edit; and the date of measurement.¶

A result is a statement about one version of one subsystem in one configuration. Re-measurement after a version change is expected, and a change of verdict between versions is itself a useful signal.¶

A set of results published together SHOULD be published in a form an unrelated party can recompute: each result pinned by a digest over its canonical encoding, the set closed by a manifest of those digests, the whole anchored to a revision of the evaluator's tooling, and the decision rule for each count stated so that a printed total is one more value to compare rather than the result itself. The reference implementation in Section 14 publishes its result set this way.¶

11. Illustrative Results (Informative)

The following verdicts illustrate what the method produces on released memory subsystems. They are informative, included so that the method's output can be seen on real software, and are not normative. The rows are those of the reference implementation's results file at the revision named in Section 14, measured on 8 October 2026 with rows added or re-measured through 17 October 2026, with the SUT stopped during each edit. A reference model the implementation carries for calibration is left out. Thirteen rows make no integrity claim and eighteen do; several subsystems appear twice because one deployment choice, where the integrity head lives or what the adversary holds, changes the result and the method asks for both.¶

In Table 1, DP is the detection point of the row. A is ACCEPTED, R is REJECTED on the read path, P is REPORTED at audit, and a dash is not applicable with the reason given in the results file.¶

Versions and configurations behind the rows:¶

langgraph-sqlite, -postgres, -redis:

LangGraph SqliteSaver (langgraph-checkpoint-sqlite 3.1.1), PostgresSaver (langgraph-checkpoint-postgres 3.1.2 on PostgreSQL 16), RedisSaver (langgraph-checkpoint-redis 0.5.2 on Redis 8).¶

openai-agents-sqlite-session:

OpenAI Agents SDK SQLiteSession (openai-agents 0.20.0).¶

llamaindex-memory-sqlite:

LlamaIndex Memory on the SQLAlchemy chat store (llama-index-core 0.14.24).¶

crewai-ltm-lancedb:

CrewAI long-term memory on a LanceDB dataset (crewai 1.15.23).¶

letta-block-history:

Letta block checkpoint history (letta 0.16.8).¶

mem0-qdrant-local:

Mem0 local vector store on Qdrant (mem0ai 2.0.20, infer off).¶

openclaw-memory-md:

OpenClaw agent memory file, read through its memory_get function (openclaw 2026.10.1).¶

hermes-memory-md:

Hermes Agent memory file, read through the system-prompt snapshot (hermes-agent 0.19.0); it has no metadata layer, so T8 does not apply.¶

vertex-memory-bank:

Vertex AI Agent Engine Memory Bank, a managed store edited through its data-plane API (google-cloud-aiplatform 2.4.0); it has no record order the evaluator can edit and cannot be copied as a unit, so T4 and T9 do not apply.¶

bedrock-agentcore-events:

AWS Bedrock AgentCore Memory, event plane, us-east-1; the plane is append-and-delete with no update, so only T2, T3, and T5 apply.¶

inspeximus-default, -rcpt+dir, -rcpt+dir+home:

inspeximus 3.0.0: receipts off, which is the default; receipts on with the adversary holding the store directory; receipts on with the adversary also holding the configuration home.¶

acrf-memory-guard:

A per-record HMAC checked on read (acrf-memory-guard 0.1.0).¶

langgraph-ledger:

A hash-chained ledger kept beside SqliteSaver, with a verify_thread audit (langgraph-ledger 0.3.0).¶

memory-blackbox-md, -md+restart:

A digest watcher over a memory file backed by a signed ledger (memory-blackbox 0.1.1): with the watching process alive across the edit, and with the process restarted before the scan.¶

atelya-attest-chain, -chain+anchor:

A keyed hash chain over the memory log (atelya-attest 0.1.1): the chain alone, and the chain with its head anchored outside the store.¶

continuum-events, -events+attest:

A hash-chained event log (continuum-agent 0.1.0): the chain alone, and the chain with an Ed25519-signed head.¶

atmem-chain, -chain+checkpoint:

An audit chain (atmem 2.3.8, wheel SHA-256 prefix 05ca2c57): the chain alone, and the chain with an external checkpoint outside the directory the adversary holds.¶

hdp-token-content-ref, -token-uuid:

A signed-token reference store (@helixar_ai/hdp 0.2.2 at b753ba9, Node 22), resolved by content reference and by UUID.¶

agent-memory:

A reference runtime over a SQLite canonical substrate with bucketed row digests and a fail-closed open (agent-memory-reference 0.2.0 at f2aef57).¶

recallspection-exact, -exact+anchor:

ExactMemory, a MAC over the whole container with a tag on every record (exactmemory.recallspection 3.2.0 at c62ef7a): the anchor beside the store, which is the product default, and the anchor off the store.¶

surgical-devops-g7:

Surgical DevOps G7 directory of signed, position-bound envelopes with a whole-store root tracked against an independent monotonic witness (surgical-dev-ops 2.6.0-rc.6 at f267cf38, Node v24.18.0).¶

Five patterns follow from the table.¶

First, the thirteen rows that make no integrity claim serve every applicable edit as genuine, the two managed services included. The read path trusts the store.¶

Second, T9 is the edit almost nothing survives. Of the twenty-nine rows where it applies, twenty-one serve the older copy as current, six notice it only at audit, and two refuse it on the read path. Four subsystems appear in the table in two configurations that differ only in where the integrity head lives, and in every one of them T9 changes: atelya-attest and continuum-agent move from ACCEPTED to REPORTED when the head is anchored or signed outside the store, and T2 moves with it, since a chain alone cannot see its own tail removed; atmem moves T9 alone with an external checkpoint; and ExactMemory moves T9 alone from ACCEPTED to REJECTED when its anchor is taken off the store. The same code gives a different answer, and the difference is a deployment decision.¶

Third, what the adversary holds matters as much as what the store checks. inspeximus with receipts on reports all nine edits at audit while the adversary holds only the store directory, and accepts T2 and T9 once the adversary also holds the configuration home. The memory-blackbox 0.1.1 fix described in the previous revision of this document holds for T1 to T8 across a restart, but a whole-store rollback that carries the ledger back with the file is accepted after a restart and reported only while the process stays alive.¶

Fourth, a per-record authenticator still catches edits to a record's bytes and nothing about where the record sits or how many there are. acrf-memory-guard rejects T1, T5, and T8 and accepts the rest. The signed-token store behaves the same way from the other side: it refuses the six edits that change or move a token, under either reference, and serves the three that remove tokens or restore an older set, because a signature on a token cannot speak to a token's absence.¶

Fifth, two rows now refuse all nine edits on the read path, where the previous revision of this document had none: surgical-devops-g7, whose whole-store root is checked against a witness the store does not hold, and recallspection-exact+anchor, whose anchor sits off the store. Two more refuse T1 to T8 on the read path and accept T9, agent-memory and recallspection-exact, which is the pattern a fail-closed container check produces when its head is kept beside the container. Of the rest, acrf-memory-guard and the HDP token store refuse a subset on the read path, and every other subsystem that claims tamper evidence detects at audit, after the agent has already resumed.¶

12. Levels (Informative)

The reference implementation publishes a ladder of five levels a subsystem can claim from its row, each a set of edits it REJECTS on the read path, in the order an adversary with store access reaches for them. The levels are cumulative and are not part of the metric; they are offered because a single metric value hides which edits a subsystem refuses, and a vendor's claim is easier to check when it names a level.¶

L0 Measured:

the subsystem has a published row, with any verdicts.¶

L1 Bytes bound:

rejects on the read path every edit that changes or adds bytes: T1, T3, T5.¶

L2 Sequence bound:

also rejects deletion, reordering, and rollback of a record: T2, T4, T7.¶

L3 Context bound:

also rejects a record moved between owners and a metadata change: T6, T8.¶

L4 Head anchored off the store:

also rejects a rollback of the whole store to an older genuine copy: T9.¶

On the rows in Table 1, surgical-devops-g7 and recallspection-exact+anchor reach L4, agent-memory and recallspection-exact reach L3, and no released subsystem sits at L1 or L2: the subsystems that bind bytes on the read path bind sequence and context as well, and the two that refuse on the read path without reaching L1 both serve a record's removal. The implementation also names a fifth level, verifiable by a third party without the store's cooperation, marked proposed until a subsystem earns it.¶

13. Design Rationale (Informative)

Four choices in this method are worth stating explicitly.¶

First, nine edits rather than one. A single "tamper the file" case lets a subsystem with a whole-store checksum pass while it still accepts rollback and cross-context replay. T6 and T7 in particular exist because encryption alone accepts them: they move only genuine bytes.¶

Second, the verdict comes from the SUT, not from the evaluator. Inspecting the store to decide whether a tamper "should" have been caught makes the result an opinion. Reading through the SUT's own read path and recording what it did makes it a measurement.¶

Third, control C1 is not optional. It is the only thing that prevents a subsystem that rejects all reads from scoring a perfect result.¶

Fourth, the SUT is stopped during the edit. The memory-blackbox result reported in the previous revision of this document is the reason: the same subsystem, the same eight edits, and the same scan went from eight of eight reported to none of eight on nothing but a restart. A method that left the attacker position to the evaluator would have produced both results and called both correct.¶

Fifth, T9 is a case of its own rather than a variant of T2. The two part ways in Table 1: atmem's chain alone reports T2 and accepts T9, and ExactMemory's container MAC rejects T2 and accepts T9, because the rollback restores the chain head or the anchor along with the records. In each of the four subsystems measured with the head beside the records and again with it outside, moving the head is what turns T9 into a pass. A method without T9 would have scored the beside-the-records configurations as tamper evident against the one edit a restored backup makes.¶

14. Reference Implementation (Informative)

An open, MIT-licensed implementation of this method, called agmi (Agent Memory Integrity), runs offline and applies the nine edits and the three controls to the memory components of the subsystems in Table 1 through a common adapter interface, with every verdict pinned to the measured version by a test that fails when the version moves. It also runs as a continuous-integration check that fails a build when any edit is accepted; one of the measured vendors has adopted it in that form, and another contributed its own adapter as a pull request that was reproduced independently before it was merged. It is provided for reproducibility and is not required to use the method.¶

The implementation's own continuous integration re-measures every row it can reach on each change and weekly against the latest releases, and fails if this measurement disagrees with the committed results on any cell both measured. The results file is published with the record described in Section 10: a digest per row over its canonical encoding, a manifest of those digests with named rejection grounds, an independent recount with its decision rule, and the git commit and the package release as anchor, together with a verifier that recomputes all of it from the pinned bytes. Two rows re-measured two days apart reproduced cell for cell, detail strings included.¶

• Code and results: https://github.com/tech4biz-yasha/agent-memory-integrity, revision 2918011 (version 0.6.3)¶
• Results site: https://agentmemoryintegrity.org¶
• Method paper: https://doi.org/10.5281/zenodo.22995111¶

15. Security Considerations

This document defines a benchmarking method for laboratory use. As with other benchmarking methodologies, the procedures here are intended for an isolated test bed and MUST NOT be run against production systems or systems the evaluator is not authorised to test. The test cases apply adversarial edits to a store; those edits MUST be confined to test data in a controlled environment.¶

The method measures a security-relevant property, whether an agent detects at-rest tampering with its memory, but a metric value from this method is not by itself an assurance of security. A high value on this method does not measure confidentiality, input-path poisoning resistance, or any property outside Section 2.¶

Publishing per-case verdicts for named subsystems and versions describes weaknesses in released software. The intent is constructive: results are stated with the version measured so that vendors can reproduce and fix them, and so that a later re-measurement shows the change.¶

16. IANA Considerations

This document has no IANA actions.¶

Acknowledgements

The author thanks the maintainers of the memory subsystems measured during the development of this method for their engagement on the individual findings, in particular the maintainers of inspeximus, who found the no-op cells that led to control C3, and of memory-blackbox, who reproduced and fixed the restart result within a day; of Surgical DevOps, who contributed their store's adapter; of Recallspection and HDP, who reviewed the description of their own systems before their rows were published, HDP having also added the measured cases to its own conformance suite; and of agent-evidence-vectors, whose corpus this implementation runs in its continuous integration and which runs this implementation's verifier in return. Thanks also to the authors of draft-han-bmwg-agent-security-benchmark for the discussion on the working group list that shaped the detection-point distinction.¶

Change Log

This section is to be removed before publication as an RFC.¶

Author's Address