miracache by MIRASTACK LABS · Open source, Apache-2.0 · Hardware partner program

Give your DDR4 servers a second life in AI inference.

miracache turns the DRAM of older servers into a shared memory tier for LLM inference, reached over RDMA. It is built to plug into vLLM and SGLang, so they can keep conversation state instead of computing it again. We are starting the first hardware tests now. Bring your hardware and run them with us.

Validated:

  • soft-RoCE
  • soft-iWARP
  • vLLM and SGLang plug-ins contract-tested against pinned engine versions

Pending:

  • hardware NICs
  • GPU serving runs
  • performance figures

01 — The problem

GPUs keep redoing work they already did

While a model reads a conversation it builds a KV cache: a saved copy of its working state for every token. That cache lives in GPU memory, which is small and expensive. When many users are active, the engine drops the cache of idle sessions. When one of those users sends the next message, the GPU reads the whole conversation again before it can answer.

What happensWhat it costs
The first word of an answer is slowRe-reading a 50,000–130,000-token conversation takes seconds
The GPU repeats finished workThat time is not available to other users
Fewer users fit on each GPUThe response-time target is missed sooner

Long agent, coding and tool-calling sessions are hit hardest: they send a long, growing history on every turn, often after a pause.

02 — What miracache does

A memory tier made of servers you already own

  • Memory servers hold the cache

    Ordinary servers with no GPU and plenty of DRAM keep the KV cache of idle sessions. Older DDR4 machines are a good fit: memory capacity and the network card matter, CPU speed barely does.

  • RDMA brings each cached block back in under a millisecond to a few milliseconds, and a long conversation in a second or two.

    Planning target The GPU server reads cached blocks straight from memory-server DRAM over a 100 GbE RDMA network (RoCE v2 or iWARP). The memory server’s CPU does no work on a read.

  • Zero changes to vLLM or SGLang

    Both load miracache as a plug-in through configuration flags only. The host check confirms the flags before a host joins.

  • A miss means recompute

    With the required settings (checked by the host check), a missing or late block is a miss: the engine recomputes it. A miss costs time.

03 — One turn, step by step

How a conversation turn flows

  • RoCE v2 · ConnectX
  • iWARP · E810

Each server needs one RDMA card: RoCE v2 or iWARP. Either protocol works.

  1. Ask. The GPU server asks the memory servers which parts of this conversation they hold.
  2. Read. It reads those parts over RDMA, straight from memory-server DRAM.
  3. Continue. The engine picks up from the first part that was not cached.
  4. Save. After the answer, new parts are written back in the background.

04 — Why DDR4

The cheapest memory is the memory you already own

Memory is now the biggest cost and the biggest shortage in AI. miracache doesn’t compete for new chips. It puts DRAM that is already installed, paid for and being retired back to work.

  • The network sets the speed, not the memory

    Estimate A memory server sends data through its RDMA card at 12.5–25 GB/s. Even older DDR4 delivers about 100–200 GB/s per CPU socket, many times more than the card can carry. Faster memory would wait on the network, so for a cache tier only capacity counts.

  • Get idle state out of the GPU

    GPU memory (HBM) is the most expensive memory in the data centre. The state of an idle session doesn’t need it while it waits. miracache moves it to the cheapest memory that can still return it in milliseconds.

  • Reuse instead of buying

    Memory makers are moving production to HBM and winding down DDR4, so new DRAM is scarce and expensive. A pool built from retired servers adds capacity without buying any new memory.

  • Shared across GPU servers

    Memory added to one GPU server serves only that server. A pool serves every GPU server on the network, and a session that moves to another GPU server finds its state there.

What a terabyte costs

Our research, November 2026, 64 GB server modules; prices change fast

MemoryApprox. cost per TB
Used DDR4 server memory$4–7k
Used DDR5$14–19k
New DDR5$41–45k in stores, over $50k on the spot market

The pool pays off most when several GPU servers share it, each memory server holds 2 TB or more, or the GPU servers have no free memory slots left. For a single GPU server with free slots, adding local memory can be cheaper, and we’ll tell you so.

DDR4 is where the spare memory is today, but DDR5 servers work just as well.

Why not local NVMe?

Estimates

PropertyLocal NVMe in the GPU serverShared DRAM pool over RDMA
Read speedabout 7 GB/s per Gen4 drive, 14 GB/s per Gen5 drive12.5 GB/s per 100G port, 25 GB/s with two
Time per read60–65 microseconds idle, up to about 190 under load, plus filesystem overheada few microseconds on the network
Wearthe cache is rewritten on every turn, so drives must be sized for it (rated 1–3 full drive writes a day)none
Shared across GPU serversno, only that server sees ityes, every GPU server reads the same pool
Survives a GPU server moveno, the session starts overyes, the session finds its state
Cost per TBlowestlow with reused DDR4

Estimate A 100,000-token conversation holds 13–31 GB of state. Over one 100G RDMA link it comes back in 1–2.5 seconds. From one Gen4 NVMe drive it takes 2–4.5 seconds, which for smaller models is about as slow as computing it again. A Gen5 drive is about as fast as one 100G link, but it still serves only its own server.

NVMe still has a place, below the pool, as cheap space for sessions that have gone cold. The order is GPU memory, local DRAM, the shared DRAM pool, then NVMe.

05 — What we want to prove together

Nine outcomes, measured on real hardware

Each goal is a hypothesis until the tests pass. We compare miracache on against miracache off with the same model, GPU, engine version, workload and response-time target.

  • O1 More sessions per GPU, lower cost per session and per token Pending

    What we measureUsers per GPU at a fixed first-token response-time (TTFT) target (p95 within the target); cost per 1 M tokens.

    Planning target≥ 1.5× users per GPU at the same target, and lower cost per 1 M tokens after adding the memory-server cost.

    StatusPending: not measured yet

  • O2 Older DDR4 servers can be repurposed for AI Pending

    What we measureCost per cached GB and per session served; hit rate and users per GPU against pool size; 3 vs 4 vs more memory servers.

    Planning targetUsers per GPU rises with pool size until the working set fits; DDR4 servers reach ≥ 90 % of the fetch rate of the best server in the pool.

    StatusPending: not measured yet

  • O3 Session failover between GPU servers and across vendors (NVIDIA ↔ AMD) on one fabric Pending Experimental

    What we measureResumed-turn TTFT against a full recompute; output correctness.

    Planning targetResumed TTFT ≤ 50 % of the cold-recompute TTFT; identical greedy outputs (temperature 0) to a no-failover control.

    ConditionsBoth GPU servers run the same model and weights revision, the same engine and engine version, the same tensor-parallel size and the same settings. Across vendors the KV cache must be BF16: NVIDIA and AMD MI300-class GPUs store FP8 in different formats. We are looking for partners with mixed NVIDIA and AMD fleets for this track.

    StatusPending: not measured yet

  • O4 Faster first token on long-context and multi-step tool-calling workloads Pending

    What we measureTTFT p50 and p99 against context length and turn number.

    Planning targetTTFT p50 reduced ≥ 2× at ≥ 32 K tokens of reused prefix; p99 not worse than off at any length.

    StatusPending: not measured yet

  • O5 Lower energy per token Pending

    What we measureGPU board power × time; joules per output token.

    Planning target≥ 20 % fewer joules per output token at the same load, including memory-server power.

    StatusPending: not measured yet

  • O6 Simpler networks: iWARP on plain Ethernet (no PFC) is good enough Pending

    What we measureFetch rate and TTFT, iWARP against RoCE v2.

    Planning targetiWARP fetch rate ≥ 85 % of RoCE v2 and TTFT p50 within 10 %.

    StatusPending: not measured yet

  • O7 Warm restart / rolling upgrade Pending

    What we measureTime to steady-state hit rate; TTFT after an engine restart.

    Planning targetHit rate back to ≥ 90 % of steady state within 2 minutes; first-turn TTFT after restart within 20 % of steady state.

    StatusPending: not measured yet

  • O8 Network sizing: bandwidth per GPU Pending

    What we measureFetch rate and TTFT at 100G vs 200G and 1 vs 2 ports; incast headroom.

    Planning targetA per-GPU-node bandwidth figure (Gb/s per GPU) at which TTFT stops improving, with zero drops on the lossless class under incast.

    StatusPending: not measured yet

  • O9 Fair sharing between deployments Pending

    Each deployment sharing the pool has its own memory cap, so a heavy deployment pushes out only its own old data, never another’s.

    What we measureA light deployment’s hit rate while a heavy one fills the pool.

    Planning target≥ 90 % of its solo hit rate.

    StatusPending: not measured yet

06 — What to bring

One rack is enough

Minimum setup: one GPU machine, three memory servers and one 100 GbE RDMA switch, in one rack.

  • Memory servers (3 or more)

    x86 servers, Intel Xeon Scalable 1st gen or newer, or AMD EPYC 7001 or newer; 1–2 sockets. DDR4 or DDR5, at least 384 GB per server (512–768 GB preferred) in 64 GB RDIMMs, one DIMM in every channel. One free x16 PCIe slot, Gen3 or newer, on the CPU that holds the memory (two if you bring both a RoCE and an iWARP card). A Gen3 slot carries one 100G port at full speed.

  • RDMA network cards

    First to be tested: NVIDIA ConnectX-6 Dx and ConnectX-7 (RoCE v2), Intel E810 (RoCE v2 and iWARP). Also welcome: Broadcom Thor2, AMD Pollara, Chelsio T6/T7 and other RoCE v2 or iWARP cards. They run on safe generic settings until tested; partners who bring them help us add tuned support. One card per server is enough. Bring both kinds only if you want the RoCE v2 vs iWARP comparison.

  • Switch

    One 100 GbE switch with 16 or more 100G ports, jumbo frames (MTU 9216). For RoCE v2: PFC on one priority and ECN, or Zero-Touch RoCE (ConnectX-6 Dx and newer are profiled for it, not yet verified on hardware). iWARP needs no PFC.

  • GPU machine (1 or more)

    NVIDIA (L40S, A100, H100, H200, B200, B300, RTX PRO 6000, DGX Spark) or AMD (MI210, MI300X, MI325X, MI355X), with an RDMA NIC near the GPUs and spare CPU cores. For the failover track: one NVIDIA and one AMD machine.

Operations: Ubuntu 24.04 LTS, a separate management network, BMC/IPMI access, SSH with sudo for our engineers.

Not sure your hardware fits? Send what you have. We test what partners bring and add each verified model to the supported list.

07 — Test setups · Option A

NVIDIA GPU server

Every setup has the same memory pool: three memory servers behind one 100 GbE RDMA switch. They differ in the GPU machine. One of options A, B or C is enough; option H adds an AMD node to an NVIDIA machine for the cross-vendor failover test.

  • RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP)
GPU machine
A 2-GPU server (L40S, A100, H100, RTX PRO 6000 or larger) or an 8-GPU node (RTX PRO 6000, H200, B200, B300).
RDMA network
One RDMA card or port to the switch: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP).
Notes
The 2-GPU server runs a small model, the 8-GPU node a large one.

07 — Test setups · Option B

NVIDIA DGX Spark (1 or 2)

  • RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP)
GPU machine
One or two DGX Spark: 128 GB unified memory each, one GB10 GPU.
RDMA network
The Spark’s built-in ConnectX-7, RoCE v2.
Notes
One Spark runs a small model (about 65 GB of weights); two split a model of up to about 200 GB.

07 — Test setups · Option C

AMD GPU node

  • RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP)
GPU machine
A large model on 4 or 8 GPUs (MI300X, MI325X, MI355X), or a small model on 2 × MI210.
RDMA network
One NIC in RoCE v2 mode: ConnectX-7, AMD Pollara 400 or Broadcom Thor2, at 100 or 200 GbE.
Notes
Pollara and Thor2 run on generic settings until tested.

07 — Test setups · Option H

Hybrid: NVIDIA and AMD

  • RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP)
GPU machines
One NVIDIA machine (option A or B) and the AMD node (option C).
RDMA network
Both machines on the same switch and the same memory pool, each with one RDMA card.
Notes
The same model and tensor-parallel size on both, and a BF16 KV cache (FP8 formats differ between vendors).

08 — How a test runs

From intake to report

Where we are: the memory server, the RDMA transport, encryption and the vLLM and SGLang plug-ins are built and tested on software RDMA, under systemd and in Docker (Docker with anonymous memory). The plug-ins are checked against the real engine code, but not yet with a GPU. Nothing has run on real RDMA network cards or GPUs yet: that is what these tests are for. They start with the network and the transport, then the engines.

  1. Intake. You tell us what hardware you have; we agree the setup, the goals to test and the dates.
  2. Host check. A read-only script reports each server’s RDMA NICs and firmware version, hugepages, memory-lock limits and IOMMU setting. We agree any changes with you before they are made.
  3. Install. miracache runs as ordinary system services, or as Docker containers. It needs permission to lock memory and a hugepage mount (set up once), and no root at run time.
  4. Network baseline. Raw RDMA read speed between every pair of servers, so the results have a fair reference.
  5. Engine tests. vLLM, then SGLang, with miracache on and off, on the same model and workload.
  6. Report. You get the full results. Anything we publish uses your name only with your approval.

08 — How a test runs: the schedule

A planned 30 days

What we test: 13 network tests, 6 service tests and the nine outcome tests (O1–O9), each run at least 3 times.

Week 1days 1–7Week 2days 8–14Week 3days 15–21Week 4days 22–30123456789101112131415161718192021222324252627282930

Hover, focus or tap a bar or a gate for details.

Planned: durations and gates are planning targets, not promises.

09 — What you get

What partners get

  • A results report from your own hardware

    Response time, users per GPU, cost per token, and read speed against raw RDMA.

  • Named credit

    In published benchmarks and papers, only with your approval.

  • A place on the supported-hardware list

    Your servers, NICs and GPUs are listed as verified once they pass.

  • Pilot support

    Hands-on help to run a production pilot after the test.

10 — Safe to run

Built to be safe on your machines

  • Encrypted end to end.Cache data is encrypted and authenticated on the GPU server before it leaves. Memory servers only ever hold ciphertext and hold no keys. Encryption cannot be turned off.
  • Isolated like the engine.Two requests share cache only when the engine itself would share it. Each session is isolated when it carries its own cache salt.
  • Least privilege.Services run as an unprivileged user. miracache never changes NIC firmware on its own.
  • Fails safe.With the required settings (checked by the host check), a missing or late block is a miss: the engine recomputes it. A memory server going down is a miss, not an error.
  • Open source.miracache is released under Apache-2.0. Read the code on GitHub.

11 — Questions

Questions

Do we need to change vLLM or SGLang?

No. Zero changes to either engine: both load miracache as a plug-in through configuration flags only.

Do we need a lossless (PFC) network?

No. iWARP runs on plain Ethernet. ConnectX-6 Dx and newer are profiled for Zero-Touch RoCE (not yet verified on hardware). If you control your switch, PFC with ECN is supported too.

Can we use InfiniBand?

Not yet. miracache runs on RoCE v2 and iWARP over Ethernet. ConnectX cards that also support Ethernet work when switched to Ethernet mode on an Ethernet switch.

Which model will we run?

The model is chosen by the GPUs you bring. Every on-versus-off comparison uses the same model and settings on both sides.

Do the memory servers need GPUs or fast CPUs?

No. They serve reads with no CPU work on the data. Memory capacity and the network card matter most.

Why not cache on local NVMe instead?

NVMe is cheaper per TB but slower per read, wears out under constant rewrites, and only its own server can use it. The shared DRAM pool is faster, serves every GPU server and keeps a session’s state when it moves. NVMe works well as a lower tier for cold sessions.

How do I install it?

Start with the README on GitHub.

What if our servers are PCIe Gen3?

They are welcome. A Gen3 x16 slot carries one 100G port at full speed.

12 — Get in touch

Bring your hardware

Tell us what you have. We reply with a proposed setup and test plan.

Thanks. We’ll reply within a few working days with a proposed setup.

Goals you want to test

* required