Note: valkey-large-object is at version 0.x. Commands, arguments, replies, and configuration can change before version 1.0.
The valkey-large-object module (BSD-3-Clause) adds a data type and commands to store large binary values in Valkey. Typical values are blocks of a machine learning KV cache, tensors, embeddings, and other blobs that are megabytes or larger in size. The module keeps objects in DRAM, or in a tiered layout where NVMe holds each object and DRAM is a read cache. Clients can transfer values over the normal Valkey connection (TCP), or over RDMA with libfabric. With RDMA, the server writes object bytes directly into client memory and reads them directly from client memory, for example with an Elastic Fabric Adapter (EFA) (see Providers).
The module registers with the name largeobj. Its value
type is largeob-k.
The largeobj.operating-mode configuration selects one of
two modes when the module loads.
Dram (default) - All objects are in a DRAM pool. The
pool is a set of pre-allocated segments. The mode does no disk I/O.Tiered - The module writes each object to a file on
NVMe in largeobj.disk-dir. DRAM is a read cache. By
default, the first read of an object promotes it into DRAM (see
largeobj.promote-min-hits and
largeobj.max-promote-size). When the cache is full, the
module drops cached copies with the lowest LFU score. The module uses
io_uring with O_DIRECT for disk I/O.The module allocates memory in fixed-size segments
(largeobj.segment-size, from 1MB to 1GB). In
Dram mode, an object must fit in a single segment. Thus an
object of largeobj.max-object-size bytes, plus a small
overhead for each chunk, must fit in one segment, and the largest object
in Dram mode is less than 1GB. The module also divides
objects into chunks (largeobj.chunk-size) for streaming
I/O. Each chunk adds a small allocation overhead in the segment. Each
pool has at most 16384 segments, so the DRAM pool does not grow past
16384 x largeobj.segment-size (1TB with 64MB segments),
also when maxmemory is larger.
The default largeobj.max-object-size is 512MB. The
smallest object is 1 byte. See Cross-configuration
rules for the rules that relate these values.
BLOB.TCP_SET and
BLOB.TCP_GET
send the value in the RESP command and reply. These commands are always
available and work with any Valkey client.BLOB.RDMA_SET and
BLOB.RDMA_GET
send only the key and a list of client memory regions. The server reads
or writes the value bytes with libfabric. The client must first open a
session with BLOB.RDMA_HELLO.If the fabric does not start, the module still loads, and only the TCP commands are available. See Valkey Large Object - RDMA Transfers for the full RDMA model.
The server calculates a CRC32C (Castagnoli) checksum of each object
when it stores the object. BLOB.INFO and BLOB.RDMA_GET
return this value. On BLOB.RDMA_SET, a
client can send its own CRC32C value. The server then rejects the write
if the checksum of the transferred bytes is different.
BLOB.TCP_SET -
stores an object that is sent in the command.BLOB.TCP_GET -
returns an object as a bulk string.BLOB.RDMA_HELLO
- opens an RDMA session on the connection.BLOB.RDMA_SET -
stores an object that the server reads from client memory.BLOB.RDMA_GET -
writes an object into client memory.BLOB.INFO -
returns the length, CRC32C, and storage tier of an object.See the complete list of large object commands.
Core commands also operate on large object keys. DEL,
UNLINK, overwrite, expiry, and eviction free the object
storage in the background. See Core commands on large object
keys. MEMORY USAGE counts the payload only when the
payload is in DRAM.
When notify-keyspace-events includes module key type
events (d), the module sends largeobj.create
when a write creates a key and largeobj.update when a write
replaces a value.
The current version has these limitations:
Tiered mode,
the module deletes largeobj.disk-dir at startup and
shutdown (see Disk
directory). See Persistence
behavior.BLOB.TCP_SET, BLOB.TCP_GET,
BLOB.RDMA_SET, and BLOB.RDMA_GET are blocking
commands, so they cannot run in transactions, scripts, functions, or
module calls.MODULE UNLOAD.
Restart the server to remove it. To stop users from using the module
without a restart, use ACLs, for example
ACL SETUSER <user> -@largeobj (see ACL).These operations work while large object keys exist, but they do not keep the object data:
SAVE, BGSAVE, SHUTDOWN with
save points, and the automatic RDB snapshots save the key names without
the object data.BGREWRITEAOF keeps no object data. With
aof-use-rdb-preamble yes (the default), the rewritten AOF
has the key names, as with SAVE. With
aof-use-rdb-preamble no, it has no large object keys.DEBUG RELOAD, or on a replica full synchronization), each
large object key reads as missing. The module starts to delete these
keys within largeobj.scaling-poll-ms (default 2.5 seconds)
after the load. With many keys, this can take longer. Until then,
EXISTS and DBSIZE still count them. For
example, EXISTS still returns 1 directly after
a BLOB.TCP_GET of the key returns nil, and returns
0 a few seconds later.DUMP returns a value without the object data.
RESTORE of that value fails with
ERR Bad data format, so MIGRATE fails (see Cluster mode).The server log shows a warning, for example
largeobj: persistence is not supported: blobs are saved to RDB/DUMP without their data,
the first time that each case occurs.
The module does not replicate large objects:
COPY from the primary of such a key
fails on the replica. The replica log then shows
This replica is sending an error to its primary: 'module key failed to copy'.The module loads on each node of a cluster. Each command uses a
single key (or no key for BLOB.RDMA_HELLO), so cluster
clients can route commands by slot.
An RDMA session belongs to one connection to one node. Send
BLOB.RDMA_HELLO on each connection to each node, and add
the server addresses of each node to the address vector. Each node
replies with its own server addresses.
You cannot move a slot that has large object keys.
MIGRATE uses DUMP and RESTORE,
which do not keep object data. The key stays on the source node:
127.0.0.1:7395> MIGRATE 127.0.0.1 7394 foo 0 5000
(error) ERR Target instance replied with error: ERR Bad data format
Atomic slot migration (CLUSTER MIGRATESLOTS) is refused
with
ERR The module largeobj does not support atomic slot migrations. Please ensure all modules have declared support for atomic slot migration and try again.
To add capacity, raise maxmemory (Dram
mode) or largeobj.disk-maxmemory (Tiered mode)
on the nodes, if the hosts have the memory or disk. Both are mutable
with CONFIG SET. Else move a slot to a new node only when
the slot has no large object keys: delete the large object keys in the
slot first (or let them expire), move the slot, and then write the
objects again. Or make a new, larger cluster and write the objects to
it.
After a MOVED or ASK redirect, the
client sends the command on a connection to the other node. If that
connection has no RDMA session, BLOB.RDMA_SET and
BLOB.RDMA_GET fail with
ERR no RDMA session (call BLOB.RDMA_HELLO first). Send
BLOB.RDMA_HELLO on that connection first.
These results are the same in Dram and
Tiered mode.
| Command | Result |
|---|---|
COPY |
Works. The copy has its own data. |
RENAME,
RENAMENX |
Work. |
MOVE,
SWAPDB |
Work. |
EXPIRE, PEXPIRE,
TTL, PERSIST |
Work. |
TYPE |
Returns largeob-k. |
OBJECT ENCODING |
Returns raw. |
DUMP |
Returns a value without the object data. |
RESTORE |
Fails with
ERR Bad data format. |
DEL, UNLINK |
Work. The module frees the storage in the background. |
FLUSHALL,
FLUSHALL ASYNC |
Work. |
INFO fields.