miracache by MIRASTACK LABS · Open source, Apache-2.0 · Hardware partner program
Give your DDR4 servers a second life in AI inference.
miracache turns the DRAM of older servers into a shared memory tier for LLM inference, reached over RDMA. It is built to plug into vLLM and SGLang, so they can keep conversation state instead of computing it again. We are starting the first hardware tests now. Bring your hardware and run them with us.
cache@mirastacklabs.ai
Validated:
- soft-RoCE
- soft-iWARP
- vLLM and SGLang plug-ins contract-tested against pinned engine versions
Pending:
- hardware NICs
- GPU serving runs
- performance figures
03 — One turn, step by step
How a conversation turn flows
- RoCE v2 · ConnectX
- iWARP · E810
Each server needs one RDMA card: RoCE v2 or iWARP. Either protocol works.
- Ask. The GPU server asks the memory servers which parts of this conversation they hold.
- Read. It reads those parts over RDMA, straight from memory-server DRAM.
- Continue. The engine picks up from the first part that was not cached.
- Save. After the answer, new parts are written back in the background.
04 — Why DDR4
The cheapest memory is the memory you already own
Memory is now the biggest cost and the biggest shortage in AI. miracache doesn’t compete for new chips. It puts DRAM that is already installed, paid for and being retired back to work.
The network sets the speed, not the memory
Estimate A memory server sends data through its RDMA card at 12.5–25 GB/s. Even older DDR4 delivers about 100–200 GB/s per CPU socket, many times more than the card can carry. Faster memory would wait on the network, so for a cache tier only capacity counts.
Get idle state out of the GPU
GPU memory (HBM) is the most expensive memory in the data centre. The state of an idle session doesn’t need it while it waits. miracache moves it to the cheapest memory that can still return it in milliseconds.
Reuse instead of buying
Memory makers are moving production to HBM and winding down DDR4, so new DRAM is scarce and expensive. A pool built from retired servers adds capacity without buying any new memory.
Shared across GPU servers
Memory added to one GPU server serves only that server. A pool serves every GPU server on the network, and a session that moves to another GPU server finds its state there.
What a terabyte costs
Our research, November 2026, 64 GB server modules; prices change fast
| Memory | Approx. cost per TB |
|---|---|
| Used DDR4 server memory | $4–7k |
| Used DDR5 | $14–19k |
| New DDR5 | $41–45k in stores, over $50k on the spot market |
The pool pays off most when several GPU servers share it, each memory server holds 2 TB or more, or the GPU servers have no free memory slots left. For a single GPU server with free slots, adding local memory can be cheaper, and we’ll tell you so.
DDR4 is where the spare memory is today, but DDR5 servers work just as well.
Why not local NVMe?
Estimates
| Property | Local NVMe in the GPU server | Shared DRAM pool over RDMA |
|---|---|---|
| Read speed | about 7 GB/s per Gen4 drive, 14 GB/s per Gen5 drive | 12.5 GB/s per 100G port, 25 GB/s with two |
| Time per read | 60–65 microseconds idle, up to about 190 under load, plus filesystem overhead | a few microseconds on the network |
| Wear | the cache is rewritten on every turn, so drives must be sized for it (rated 1–3 full drive writes a day) | none |
| Shared across GPU servers | no, only that server sees it | yes, every GPU server reads the same pool |
| Survives a GPU server move | no, the session starts over | yes, the session finds its state |
| Cost per TB | lowest | low with reused DDR4 |
Estimate A 100,000-token conversation holds 13–31 GB of state. Over one 100G RDMA link it comes back in 1–2.5 seconds. From one Gen4 NVMe drive it takes 2–4.5 seconds, which for smaller models is about as slow as computing it again. A Gen5 drive is about as fast as one 100G link, but it still serves only its own server.
NVMe still has a place, below the pool, as cheap space for sessions that have gone cold. The order is GPU memory, local DRAM, the shared DRAM pool, then NVMe.
05 — What we want to prove together
Nine outcomes, measured on real hardware
Each goal is a hypothesis until the tests pass. We compare miracache on against miracache off with the same model, GPU, engine version, workload and response-time target.
-
O1 More sessions per GPU, lower cost per session and per token Pending
What we measureUsers per GPU at a fixed first-token response-time (TTFT) target (p95 within the target); cost per 1 M tokens.
Planning target≥ 1.5× users per GPU at the same target, and lower cost per 1 M tokens after adding the memory-server cost.
StatusPending: not measured yet
-
O2 Older DDR4 servers can be repurposed for AI Pending
What we measureCost per cached GB and per session served; hit rate and users per GPU against pool size; 3 vs 4 vs more memory servers.
Planning targetUsers per GPU rises with pool size until the working set fits; DDR4 servers reach ≥ 90 % of the fetch rate of the best server in the pool.
StatusPending: not measured yet
-
O3 Session failover between GPU servers and across vendors (NVIDIA ↔ AMD) on one fabric Pending Experimental
What we measureResumed-turn TTFT against a full recompute; output correctness.
Planning targetResumed TTFT ≤ 50 % of the cold-recompute TTFT; identical greedy outputs (temperature 0) to a no-failover control.
ConditionsBoth GPU servers run the same model and weights revision, the same engine and engine version, the same tensor-parallel size and the same settings. Across vendors the KV cache must be BF16: NVIDIA and AMD MI300-class GPUs store FP8 in different formats. We are looking for partners with mixed NVIDIA and AMD fleets for this track.
StatusPending: not measured yet
-
O4 Faster first token on long-context and multi-step tool-calling workloads Pending
What we measureTTFT p50 and p99 against context length and turn number.
Planning targetTTFT p50 reduced ≥ 2× at ≥ 32 K tokens of reused prefix; p99 not worse than off at any length.
StatusPending: not measured yet
-
O5 Lower energy per token Pending
What we measureGPU board power × time; joules per output token.
Planning target≥ 20 % fewer joules per output token at the same load, including memory-server power.
StatusPending: not measured yet
-
O6 Simpler networks: iWARP on plain Ethernet (no PFC) is good enough Pending
What we measureFetch rate and TTFT, iWARP against RoCE v2.
Planning targetiWARP fetch rate ≥ 85 % of RoCE v2 and TTFT p50 within 10 %.
StatusPending: not measured yet
-
O7 Warm restart / rolling upgrade Pending
What we measureTime to steady-state hit rate; TTFT after an engine restart.
Planning targetHit rate back to ≥ 90 % of steady state within 2 minutes; first-turn TTFT after restart within 20 % of steady state.
StatusPending: not measured yet
-
O8 Network sizing: bandwidth per GPU Pending
What we measureFetch rate and TTFT at 100G vs 200G and 1 vs 2 ports; incast headroom.
Planning targetA per-GPU-node bandwidth figure (Gb/s per GPU) at which TTFT stops improving, with zero drops on the lossless class under incast.
StatusPending: not measured yet
-
O9 Fair sharing between deployments Pending
Each deployment sharing the pool has its own memory cap, so a heavy deployment pushes out only its own old data, never another’s.
What we measureA light deployment’s hit rate while a heavy one fills the pool.
Planning target≥ 90 % of its solo hit rate.
StatusPending: not measured yet
06 — What to bring
One rack is enough
Minimum setup: one GPU machine, three memory servers and one 100 GbE RDMA switch, in one rack.
Memory servers (3 or more)
x86 servers, Intel Xeon Scalable 1st gen or newer, or AMD EPYC 7001 or newer; 1–2 sockets. DDR4 or DDR5, at least 384 GB per server (512–768 GB preferred) in 64 GB RDIMMs, one DIMM in every channel. One free x16 PCIe slot, Gen3 or newer, on the CPU that holds the memory (two if you bring both a RoCE and an iWARP card). A Gen3 slot carries one 100G port at full speed.
RDMA network cards
First to be tested: NVIDIA ConnectX-6 Dx and ConnectX-7 (RoCE v2), Intel E810 (RoCE v2 and iWARP). Also welcome: Broadcom Thor2, AMD Pollara, Chelsio T6/T7 and other RoCE v2 or iWARP cards. They run on safe generic settings until tested; partners who bring them help us add tuned support. One card per server is enough. Bring both kinds only if you want the RoCE v2 vs iWARP comparison.
Switch
One 100 GbE switch with 16 or more 100G ports, jumbo frames (MTU 9216). For RoCE v2: PFC on one priority and ECN, or Zero-Touch RoCE (ConnectX-6 Dx and newer are profiled for it, not yet verified on hardware). iWARP needs no PFC.
GPU machine (1 or more)
NVIDIA (L40S, A100, H100, H200, B200, B300, RTX PRO 6000, DGX Spark) or AMD (MI210, MI300X, MI325X, MI355X), with an RDMA NIC near the GPUs and spare CPU cores. For the failover track: one NVIDIA and one AMD machine.
Operations: Ubuntu 24.04 LTS, a separate management network, BMC/IPMI access, SSH with sudo for our engineers.
Not sure your hardware fits? Send what you have. We test what partners bring and add each verified model to the supported list.
08 — How a test runs
From intake to report
Where we are: the memory server, the RDMA transport, encryption and the vLLM and SGLang plug-ins are built and tested on software RDMA, under systemd and in Docker (Docker with anonymous memory). The plug-ins are checked against the real engine code, but not yet with a GPU. Nothing has run on real RDMA network cards or GPUs yet: that is what these tests are for. They start with the network and the transport, then the engines.
- Intake. You tell us what hardware you have; we agree the setup, the goals to test and the dates.
- Host check. A read-only script reports each server’s RDMA NICs and firmware version, hugepages, memory-lock limits and IOMMU setting. We agree any changes with you before they are made.
- Install. miracache runs as ordinary system services, or as Docker containers. It needs permission to lock memory and a hugepage mount (set up once), and no root at run time.
- Network baseline. Raw RDMA read speed between every pair of servers, so the results have a fair reference.
- Engine tests. vLLM, then SGLang, with miracache on and off, on the same model and workload.
- Report. You get the full results. Anything we publish uses your name only with your approval.
08 — How a test runs: the schedule
A planned 30 days
What we test: 13 network tests, 6 service tests and the nine outcome tests (O1–O9), each run at least 3 times.
Hover, focus or tap a bar or a gate for details.
Planned: durations and gates are planning targets, not promises.