Multi-GPU: Tensor, Pipeline and Hybrid Parallelism

This vignette covers running ggmlR across more than one GPU. For the single-GPU basics — installation, device discovery, Vulkan build flags — see vignette("gpu-vulkan"). For data-parallel training specifically, see vignette("data-parallel-training").

1. Which mode do you need?

The three modes differ in what gets split across devices, and that choice follows from why one GPU is not enough.

Mode	What is split	Use when	Entry point
Tensor parallelism (TP)	Individual weight matrices, by rows	One layer is too big, or you want to cut per-layer latency	`ggml_vulkan_split_mul_mat()`
Pipeline parallelism (PP)	Whole layers, by depth	The model does not fit on one card	`ggml_pp_forward()`
Data parallelism (DP)	The batch	The model fits, you want throughput	`dp_train()`, `ggml_tp_dp_forward()`

They compose. ggml_tp_dp_forward() and ggml_pp_dp_forward() run DP replicas on top of TP or PP groups — for example four GPUs as two replicas of TP=2.

The decisive practical difference is how much data crosses between devices:

TP moves activations across devices twice per split layer.
PP moves activations across devices once per stage boundary — for a 2-stage split of a 32-layer model, that is one transfer per forward pass.
DP moves nothing between replicas during inference.

That ordering matters more than it looks, because of the transport.

Are you sure you want this layer?

ggmlR provides the tensor layer: primitives that split individual matrices, graph stages and batches across devices. If your goal is to run a whole language model across several GPUs, you almost certainly want llamaR instead, which is built on ggmlR and handles model loading, KV cache, sampling and serving:

# llamaR — whole-LLM inference; the split happens inside the model loader
model <- llama_load_model("model.gguf", n_gpu_layers = -1L,
                          devices = c("Vulkan0", "Vulkan1"),
                          split_mode = "row")     # or "layer" for pipeline

The names coincide but the code does not: split_mode = "row" / "layer" is llamaR’s own weight distribution over a real transformer, not a call into the ggml_vulkan_split_mul_mat() / ggml_pp_forward() shown below. Reach for the ggmlR primitives when you are building a custom parallel computation — a model architecture of your own, a research prototype, a non-LLM workload — rather than serving an existing GGUF.

llamaR ships a full PP/TP/DP sweep over a real model in system.file("examples", "bench_pp_tp_dp.sh", package = "llamaR").

2. The transport, and why it constrains everything

Copying a tensor from GPU 0 to GPU 1 needs a path. ggmlR implements three, selected by the transport argument:

"host-staging" (default) — device → host RAM → device. Portable and always correct.
"opaque-fd" — zero-copy via VK_KHR_external_memory_fd.
"device-group" — one logical device spanning several cards (NVLink / LDA).

The default is not a conservative placeholder; on the hardware we tested it is the only path that works. Measured on a 4×P100 server:

"opaque-fd" shares memory correctly in loopback (same device) but silently transfers zeros across devices — a hardware/driver limitation on NVIDIA, not a bug in the import path.
"device-group" enumerates five groups, every one of them single-device: no NVLink/LDA aggregation is offered at all.

Verify your own hardware before assuming a zero-copy path exists:

# Loopback sanity check: does the transport work at all on device 0?
r <- ggml_vulkan_p2p_selftest(0L, 0L)
cat(r$report)

# The real question: does a cross-device copy carry the bytes?
if (ggml_vulkan_device_count() >= 2) {
  r <- ggml_vulkan_p2p_selftest(0L, 1L, transport = "opaque-fd")
  cat(r$report)          # "verified" vs. a mismatch tells you immediately
}

# Are there any multi-device (LDA) groups?
ggml_vulkan_device_groups()

Because host-staging routes through host RAM, TP is the mode that suffers most and PP the least. If your model does not fit on one card, prefer PP.

3. Tensor parallelism

ggml_vulkan_split_mul_mat() splits the weight matrix W by rows across devices, computes each slice on its own GPU, and gathers the result. It is a standalone C routine, deliberately outside the ggml graph.

set.seed(1)
W <- matrix(rnorm(2048 * 64), nrow = 2048)   # [N x K] weights
X <- matrix(rnorm(4 * 64),    nrow = 4)      # [M x K] activations

Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2)

# Contract: identical to the single-device result, up to f32 rounding.
max(abs(Y - X %*% t(W)))                     # ~3.8e-6 on 2 devices

Pick a subset of GPUs rather than “the first N” with device_ids, and give uneven cards uneven shares with weights:

Y <- ggml_vulkan_split_mul_mat(W, X, device_ids = c(0L, 1L))
Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2, weights = c(0.7, 0.3))

# Inspect the row ranges the split math will use (0-based, half-open):
ggml_vulkan_split_row_ranges(nrows = 2048, n_devices = 2)

ggml_vulkan_split_buffer_type() exposes the same row-split as a ggml buffer type, for callers that allocate weights themselves.

4. Pipeline parallelism

PP assigns layers to devices: GPU 0 runs stages 1–16, GPU 1 runs 17–32, and exactly one activation tensor crosses between them. Each stage is described by a list with its device, its in_shape, and a build function that constructs the sub-graph and returns a set_weights closure.

The one non-obvious rule: weights are set after allocation, which is why build returns set_weights instead of filling tensors immediately.

K <- 64L; M <- 8L
W1 <- matrix(rnorm(K * K), nrow = K)
W2 <- matrix(rnorm(K * K), nrow = K)
X  <- matrix(rnorm(K * M), nrow = K)     # ggml ne = c(K, M): column m is sample m

make_stage <- function(dev, Wt, relu) {
  list(
    device   = dev,
    in_shape = c(K, M),
    build = function(ctx, input) {
      w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K)
      z <- ggml_mul_mat(ctx, w, input)     # == t(Wt) %*% input in R terms
      list(
        output      = if (relu) ggml_relu(ctx, z) else z,
        set_weights = function() ggml_backend_tensor_set_data(w, as.numeric(Wt))
      )
    })
}

stages <- list(make_stage(0L, W1, relu = TRUE),
               make_stage(1L, W2, relu = FALSE))

y <- ggml_pp_forward(stages, x = as.numeric(X), out_shape = c(K, M))
Y <- matrix(y, nrow = K, ncol = M)

max(abs(Y - t(W2) %*% pmax(t(W1) %*% X, 0)))   # ~1.8e-5

5. Hybrid: DP over TP or PP

With four GPUs, two replicas of TP=2 usually beat one TP=4 group: DP adds no cross-device traffic, so the hybrid hides the weakness of host-staging.

# Replica A = {GPU0, GPU1}, replica B = {GPU2, GPU3}.
# Each replica takes half the batch; no traffic crosses between replicas.
Y <- ggml_tp_dp_forward(W, X, replicas = list(c(0L, 1L), c(2L, 3L)))

# The PP equivalent takes a factory: make_stages(devices, m_shard) -> stage list
Y <- ggml_pp_dp_forward(make_stages, x, replicas = list(c(0L, 1L), c(2L, 3L)),
                        out_ncol = M)

For multi-GPU training rather than inference, use dp_train() — see vignette("data-parallel-training").

6. Clean shutdown (and the `--enable-hard-exit` flag)

A multi-GPU script can segfault after it has printed its results. The cause is a race at process exit: the Vulkan loader and the driver’s ICD libraries get unmapped while driver worker threads are still winding down, so late destructors call into unmapped memory. The computed results are already out; the crash is harmless but noisy, and it turns a successful run into a non-zero exit code.

Call ggml_vulkan_shutdown() to tear Vulkan down while the loader is still mapped. It is idempotent, safe mid-session, and Vulkan transparently re-initializes on the next operation:

ggml_vulkan_shutdown()     # safe anywhere; releases devices

That narrows the race but does not close it: no R exit hook runs before R unmaps the loader. For a guaranteed-clean exit, make hard = TRUE the last statement of a standalone script — after teardown it calls _exit(status), terminating immediately without running exit handlers or unmapping libraries, so there is no teardown phase left to crash.

ggml_vulkan_shutdown(hard = TRUE, status = 0L)   # never returns

This path is compiled out by default. CRAN Repository Policy forbids a package from terminating the user’s R session, so the released package must not link _exit(). In a default build, hard = TRUE performs the normal teardown and emits a warning() — it is never silently ignored. Compile it in with:

R CMD INSTALL . --configure-args="--enable-hard-exit"

On Windows R ignores configure.args; set Sys.setenv(GGML_VK_HARD_EXIT = "1") before installing from source. Check what the current build has:

ggml_vulkan_hard_exit_available()

Two caveats. hard = TRUE bypasses R’s shutdown entirely: no .RData, no on.exit() handlers, no flushed connections beyond what ggmlR flushes itself. Use it only as the final line of a script, never mid-session, never inside a package. And although the race is observed in multi-GPU scripts — more devices mean more driver threads and a wider window — nothing in the mechanism is specific to multiple GPUs; single-GPU runs simply have not been seen to trip it.

7. Where to look next

Runnable demos ship with the package, under system.file("examples", package = "ggmlR"):

list.files(system.file("examples", package = "ggmlR"),
           pattern = "^(tp_|pp_|multi_gpu|dp_)")

multi_gpu_example.R — device discovery and per-device work
tp_dp_hybrid.R — TP×DP hybrid on four GPUs
pp_pipeline.R — two-stage pipeline parallelism
tp_p2p_diagnose.R — transport diagnostics; drop hard = TRUE to observe the exit-time crash described above
dp_train_demo.R — data-parallel training

The test suite documents the numeric contracts, and skips gracefully when fewer than two GPUs are present:

tests/testthat/test-vulkan-tensor-parallel.R — split math, split_mul_mat, buffer-type factory, TP×DP and PP forwards
tests/testthat/test-vulkan.R, test-vulkan-caps.R — backend and capabilities