# MIRASTACK LABS: miracache (full text) > All miracache Markdown in one file, regenerated at every build (2026-11-19). Index: https://www.mirastacklabs.ai/llms.txt --- URL: https://www.mirastacklabs.ai/miracache.md # Give your DDR4 servers a second life in AI inference. > Source: https://www.mirastacklabs.ai/miracache/ (this Markdown is generated from the page at build time, 2026-11-19; the HTML page is canonical). miracache by MIRASTACK LABS · Open source, Apache-2.0 · Hardware partner program miracache turns the DRAM of older servers into a shared memory tier for LLM inference, reached over RDMA. It is built to plug into vLLM and SGLang, so they can keep conversation state instead of computing it again. We are starting the first hardware tests now. Bring your hardware and run them with us. [Bring your hardware](https://www.mirastacklabs.ai/miracache/#contact) [Email us](https://www.mirastacklabs.ai/miracache/#email) [GitHub](https://github.com/mirastacklabs-ai/miracache) [Docs](https://github.com/mirastacklabs-ai/miracache#readme) Validated: - soft-RoCE - soft-iWARP - vLLM and SGLang plug-ins contract-tested against pinned engine versions Pending: - hardware NICs - GPU serving runs - performance figures 01 — The problem ## GPUs keep redoing work they already did While a model reads a conversation it builds a **KV cache**: a saved copy of its working state for every token. That cache lives in GPU memory, which is small and expensive. When many users are active, the engine drops the cache of idle sessions. When one of those users sends the next message, the GPU reads the whole conversation again before it can answer. | What happens | What it costs | | --- | --- | | The first word of an answer is slow | Re-reading a 50,000–130,000-token conversation takes seconds | | The GPU repeats finished work | That time is not available to other users | | Fewer users fit on each GPU | The response-time target is missed sooner | Long agent, coding and tool-calling sessions are hit hardest: they send a long, growing history on every turn, often after a pause. 02 — What miracache does ## A memory tier made of servers you already own - **Memory servers hold the cache** Ordinary servers with no GPU and plenty of DRAM keep the KV cache of idle sessions. Older DDR4 machines are a good fit: memory capacity and the network card matter, CPU speed barely does. - **RDMA brings each cached block back in under a millisecond to a few milliseconds, and a long conversation in a second or two.** Planning target The GPU server reads cached blocks straight from memory-server DRAM over a 100 GbE RDMA network (RoCE v2 or iWARP). The memory server’s CPU does no work on a read. - **Zero changes to vLLM or SGLang** Both load miracache as a plug-in through configuration flags only. The host check confirms the flags before a host joins. - **A miss means recompute** With the required settings (checked by the host check), a missing or late block is a miss: the engine recomputes it. A miss costs time. 03 — One turn, step by step ## How a conversation turn flows - RoCE v2 · ConnectX - iWARP · E810 Each server needs one RDMA card: RoCE v2 or iWARP. Either protocol works. 1. **Ask.** The GPU server asks the memory servers which parts of this conversation they hold. 2. **Read.** It reads those parts over RDMA, straight from memory-server DRAM. 3. **Continue.** The engine picks up from the first part that was not cached. 4. **Save.** After the answer, new parts are written back in the background. 04 — Why DDR4 ## The cheapest memory is the memory you already own Memory is now the biggest cost and the biggest shortage in AI. miracache doesn’t compete for new chips. It puts DRAM that is already installed, paid for and being retired back to work. - **The network sets the speed, not the memory** Estimate A memory server sends data through its RDMA card at 12.5–25 GB/s. Even older DDR4 delivers about 100–200 GB/s per CPU socket, many times more than the card can carry. Faster memory would wait on the network, so for a cache tier only capacity counts. - **Get idle state out of the GPU** GPU memory (HBM) is the most expensive memory in the data centre. The state of an idle session doesn’t need it while it waits. miracache moves it to the cheapest memory that can still return it in milliseconds. - **Reuse instead of buying** Memory makers are moving production to HBM and winding down DDR4, so new DRAM is scarce and expensive. A pool built from retired servers adds capacity without buying any new memory. - **Shared across GPU servers** Memory added to one GPU server serves only that server. A pool serves every GPU server on the network, and a session that moves to another GPU server finds its state there. ### What a terabyte costs Our research, November 2026, 64 GB server modules; prices change fast | Memory | Approx. cost per TB | | --- | --- | | Used DDR4 server memory | $4–7k | | Used DDR5 | $14–19k | | New DDR5 | $41–45k in stores, over $50k on the spot market | The pool pays off most when several GPU servers share it, each memory server holds 2 TB or more, or the GPU servers have no free memory slots left. For a single GPU server with free slots, adding local memory can be cheaper, and we’ll tell you so. DDR4 is where the spare memory is today, but DDR5 servers work just as well. ### Why not local NVMe? Estimates | Property | Local NVMe in the GPU server | Shared DRAM pool over RDMA | | --- | --- | --- | | Read speed | about 7 GB/s per Gen4 drive, 14 GB/s per Gen5 drive | 12.5 GB/s per 100G port, 25 GB/s with two | | Time per read | 60–65 microseconds idle, up to about 190 under load, plus filesystem overhead | a few microseconds on the network | | Wear | the cache is rewritten on every turn, so drives must be sized for it (rated 1–3 full drive writes a day) | none | | Shared across GPU servers | no, only that server sees it | yes, every GPU server reads the same pool | | Survives a GPU server move | no, the session starts over | yes, the session finds its state | | Cost per TB | lowest | low with reused DDR4 | Estimate A 100,000-token conversation holds 13–31 GB of state. Over one 100G RDMA link it comes back in 1–2.5 seconds. From one Gen4 NVMe drive it takes 2–4.5 seconds, which for smaller models is about as slow as computing it again. A Gen5 drive is about as fast as one 100G link, but it still serves only its own server. NVMe still has a place, below the pool, as cheap space for sessions that have gone cold. The order is GPU memory, local DRAM, the shared DRAM pool, then NVMe. 05 — What we want to prove together ## Nine outcomes, measured on real hardware Each goal is a hypothesis until the tests pass. We compare miracache on against miracache off with the same model, GPU, engine version, workload and response-time target. - **O1 More sessions per GPU, lower cost per session and per token Pending** What we measureUsers per GPU at a fixed first-token response-time (TTFT) target (p95 within the target); cost per 1 M tokens. Planning target≥ 1.5× users per GPU at the same target, and lower cost per 1 M tokens after adding the memory-server cost. StatusPending: not measured yet - **O2 Older DDR4 servers can be repurposed for AI Pending** What we measureCost per cached GB and per session served; hit rate and users per GPU against pool size; 3 vs 4 vs more memory servers. Planning targetUsers per GPU rises with pool size until the working set fits; DDR4 servers reach ≥ 90 % of the fetch rate of the best server in the pool. StatusPending: not measured yet - **O3 Session failover between GPU servers and across vendors (NVIDIA ↔ AMD) on one fabric Pending Experimental** What we measureResumed-turn TTFT against a full recompute; output correctness. Planning targetResumed TTFT ≤ 50 % of the cold-recompute TTFT; identical greedy outputs (temperature 0) to a no-failover control. ConditionsBoth GPU servers run the same model and weights revision, the same engine and engine version, the same tensor-parallel size and the same settings. Across vendors the KV cache must be BF16: NVIDIA and AMD MI300-class GPUs store FP8 in different formats. We are looking for partners with mixed NVIDIA and AMD fleets for this track. StatusPending: not measured yet - **O4 Faster first token on long-context and multi-step tool-calling workloads Pending** What we measureTTFT p50 and p99 against context length and turn number. Planning targetTTFT p50 reduced ≥ 2× at ≥ 32 K tokens of reused prefix; p99 not worse than off at any length. StatusPending: not measured yet - **O5 Lower energy per token Pending** What we measureGPU board power × time; joules per output token. Planning target≥ 20 % fewer joules per output token at the same load, including memory-server power. StatusPending: not measured yet - **O6 Simpler networks: iWARP on plain Ethernet (no PFC) is good enough Pending** What we measureFetch rate and TTFT, iWARP against RoCE v2. Planning targetiWARP fetch rate ≥ 85 % of RoCE v2 and TTFT p50 within 10 %. StatusPending: not measured yet - **O7 Warm restart / rolling upgrade Pending** What we measureTime to steady-state hit rate; TTFT after an engine restart. Planning targetHit rate back to ≥ 90 % of steady state within 2 minutes; first-turn TTFT after restart within 20 % of steady state. StatusPending: not measured yet - **O8 Network sizing: bandwidth per GPU Pending** What we measureFetch rate and TTFT at 100G vs 200G and 1 vs 2 ports; incast headroom. Planning targetA per-GPU-node bandwidth figure (Gb/s per GPU) at which TTFT stops improving, with zero drops on the lossless class under incast. StatusPending: not measured yet - **O9 Fair sharing between deployments Pending** Each deployment sharing the pool has its own memory cap, so a heavy deployment pushes out only its own old data, never another’s. What we measureA light deployment’s hit rate while a heavy one fills the pool. Planning target≥ 90 % of its solo hit rate. StatusPending: not measured yet 06 — What to bring ## One rack is enough Minimum setup: one GPU machine, three memory servers and one 100 GbE RDMA switch, in one rack. - **Memory servers (3 or more)** x86 servers, Intel Xeon Scalable 1st gen or newer, or AMD EPYC 7001 or newer; 1–2 sockets. DDR4 or DDR5, at least 384 GB per server (512–768 GB preferred) in 64 GB RDIMMs, one DIMM in every channel. One free x16 PCIe slot, Gen3 or newer, on the CPU that holds the memory (two if you bring both a RoCE and an iWARP card). A Gen3 slot carries one 100G port at full speed. - **RDMA network cards** First to be tested: NVIDIA ConnectX-6 Dx and ConnectX-7 (RoCE v2), Intel E810 (RoCE v2 and iWARP). Also welcome: Broadcom Thor2, AMD Pollara, Chelsio T6/T7 and other RoCE v2 or iWARP cards. They run on safe generic settings until tested; partners who bring them help us add tuned support. One card per server is enough. Bring both kinds only if you want the RoCE v2 vs iWARP comparison. - **Switch** One 100 GbE switch with 16 or more 100G ports, jumbo frames (MTU 9216). For RoCE v2: PFC on one priority and ECN, or Zero-Touch RoCE (ConnectX-6 Dx and newer are profiled for it, not yet verified on hardware). iWARP needs no PFC. - **GPU machine (1 or more)** NVIDIA (L40S, A100, H100, H200, B200, B300, RTX PRO 6000, DGX Spark) or AMD (MI210, MI300X, MI325X, MI355X), with an RDMA NIC near the GPUs and spare CPU cores. For the failover track: one NVIDIA and one AMD machine. **Operations:** Ubuntu 24.04 LTS, a separate management network, BMC/IPMI access, SSH with sudo for our engineers. Not sure your hardware fits? Send what you have. We test what partners bring and add each verified model to the supported list. 07 — Test setups · Option A ## NVIDIA GPU server Every setup has the same memory pool: three memory servers behind one 100 GbE RDMA switch. They differ in the GPU machine. One of options A, B or C is enough; option H adds an AMD node to an NVIDIA machine for the cross-vendor failover test. - RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP) 07 — Test setups · Option B ## NVIDIA DGX Spark (1 or 2) - RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP) 07 — Test setups · Option C ## AMD GPU node - RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP) 07 — Test setups · Option H ## Hybrid: NVIDIA and AMD - RDMA link: ConnectX (RoCE v2) or E810 (RoCE v2 or iWARP) 08 — How a test runs ## From intake to report Where we are: the memory server, the RDMA transport, encryption and the vLLM and SGLang plug-ins are built and tested on software RDMA, under systemd and in Docker (Docker with anonymous memory). The plug-ins are checked against the real engine code, but not yet with a GPU. Nothing has run on real RDMA network cards or GPUs yet: that is what these tests are for. They start with the network and the transport, then the engines. 1. **Intake.** You tell us what hardware you have; we agree the setup, the goals to test and the dates. 2. **Host check.** A read-only script reports each server’s RDMA NICs and firmware version, hugepages, memory-lock limits and IOMMU setting. We agree any changes with you before they are made. 3. **Install.** miracache runs as ordinary system services, or as Docker containers. It needs permission to lock memory and a hugepage mount (set up once), and no root at run time. 4. **Network baseline.** Raw RDMA read speed between every pair of servers, so the results have a fair reference. 5. **Engine tests.** vLLM, then SGLang, with miracache on and off, on the same model and workload. 6. **Report.** You get the full results. Anything we publish uses your name only with your approval. 08 — How a test runs: the schedule ## A planned 30 days What we test: 13 network tests, 6 service tests and the nine outcome tests (O1–O9), each run at least 3 times. Hover, focus or tap a bar or a gate for details. Planned: durations and gates are planning targets, not promises. 09 — What you get ## What partners get - **A results report from your own hardware** Response time, users per GPU, cost per token, and read speed against raw RDMA. - **Named credit** In published benchmarks and papers, only with your approval. - **A place on the supported-hardware list** Your servers, NICs and GPUs are listed as verified once they pass. - **Pilot support** Hands-on help to run a production pilot after the test. 10 — Safe to run ## Built to be safe on your machines - **Encrypted end to end.** Cache data is encrypted and authenticated on the GPU server before it leaves. Memory servers only ever hold ciphertext and hold no keys. Encryption cannot be turned off. - **Isolated like the engine.** Two requests share cache only when the engine itself would share it. Each session is isolated when it carries its own cache salt. - **Least privilege.** Services run as an unprivileged user. miracache never changes NIC firmware on its own. - **Fails safe.** With the required settings (checked by the host check), a missing or late block is a miss: the engine recomputes it. A memory server going down is a miss, not an error. - **Open source.** miracache is released under Apache-2.0. [Read the code on GitHub](https://github.com/mirastacklabs-ai/miracache). 11 — Questions ## Questions ### Do we need to change vLLM or SGLang? No. Zero changes to either engine: both load miracache as a plug-in through configuration flags only. ### Do we need a lossless (PFC) network? No. iWARP runs on plain Ethernet. ConnectX-6 Dx and newer are profiled for Zero-Touch RoCE (not yet verified on hardware). If you control your switch, PFC with ECN is supported too. ### Can we use InfiniBand? Not yet. miracache runs on RoCE v2 and iWARP over Ethernet. ConnectX cards that also support Ethernet work when switched to Ethernet mode on an Ethernet switch. ### Which model will we run? The model is chosen by the GPUs you bring. Every on-versus-off comparison uses the same model and settings on both sides. ### Do the memory servers need GPUs or fast CPUs? No. They serve reads with no CPU work on the data. Memory capacity and the network card matter most. ### Why not cache on local NVMe instead? NVMe is cheaper per TB but slower per read, wears out under constant rewrites, and only its own server can use it. The shared DRAM pool is faster, serves every GPU server and keeps a session’s state when it moves. NVMe works well as a lower tier for cold sessions. ### How do I install it? Start with the [README on GitHub](https://github.com/mirastacklabs-ai/miracache#readme). ### What if our servers are PCIe Gen3? They are welcome. A Gen3 x16 slot carries one 100G port at full speed. 12 — Get in touch ## Bring your hardware Tell us what you have. We reply with a proposed setup and test plan. Prefer email? Write to `cache@mirastacklabs.ai` with the same details. Partner form: https://www.mirastacklabs.ai/miracache/#contact (or write to cache@mirastacklabs.ai). --- URL: https://www.mirastacklabs.ai/miracache/quickstart.md # miracache: quick start > Source: https://github.com/mirastacklabs-ai/miracache/blob/main/README.md (public repo, ref main), sections "Requirements" and "Quick start"; pulled at build time 2026-11-19. The repo has no docs/quickstart.md yet; this file switches to it automatically when it appears. ## Requirements | Role | Needs | |---|---| | Memory node | x86-64 Linux (tested on Ubuntu 24.04), an RDMA NIC speaking RoCEv2 or iWARP (or soft-RoCE / soft-iWARP for trials), hugepages, plenty of DRAM. Three or more nodes | | GPU host | vLLM 0.32–0.34 (accepted range; contract-tested at 0.33.0) or SGLang 0.5.23 (contract-tested), Python 3.10+, the same RDMA fabric as the memory nodes | | Build | clang-18, CMake 3.28+, Ninja, rdma-core (`libibverbs-dev`, `librdmacm-dev`), OpenSSL (`libssl-dev`). Go is fetched and checksum-verified by `deploy/docker/go.sh` | ## Quick start **1. Build** ```sh cmake --preset release && cmake --build --preset release # mira-memnode, mira-nicprobe, libmiracache.so deploy/docker/go.sh -C controller build -o "$PWD/build/release/bin/" ./cmd/mira-controller ./cmd/mirctl # or container images: deploy/docker/build.sh ``` **2. Check each host** ```sh build/release/core/fabric/mira-nicprobe --json # RDMA devices, capabilities, matched NIC profile PYTHONPATH=python python -m miracache.node_prep --hugepage-gib 16 # dry run: hugepages, memlock, RDMA modules PYTHONPATH=python python -m miracache.node_prep --hugepage-gib 16 --apply --yes # as root, once you agree with the plan ``` Size `--hugepage-gib` to the cache you give each memory node (the example config below uses 16 GiB). **3. Start three or more memory nodes.** Start from [`deploy/examples/memnode.toml`](https://github.com/mirastacklabs-ai/miracache/blob/main/deploy/examples/memnode.toml) (addresses, device, hugepage store, certificates) and run them under systemd ([`deploy/systemd/`](https://github.com/mirastacklabs-ai/miracache/blob/main/deploy/systemd/README.md)) or Docker ([`deploy/docker/`](https://github.com/mirastacklabs-ai/miracache/blob/main/deploy/docker/README.md)). Memory nodes and clients authenticate each other with mutual TLS; use your own CA. **4. Configure the GPU hosts.** Start from [`deploy/examples/client.toml`](https://github.com/mirastacklabs-ai/miracache/blob/main/deploy/examples/client.toml) and create the deployment secret once, then copy the same file to every GPU host (never to a memory node). Run as the user the engine runs as, and keep it on persistent storage: a lost secret makes the whole cache unreadable. ```sh sudo install -d -m 0700 -o "$USER" /etc/miracache/keys umask 077; printf 'current = 1\n1 = %s\n' "$(openssl rand -hex 32)" > /etc/miracache/keys/secret ``` **5. Check the engine flags, then launch.** `mira-preflight` hard-fails a host whose engine settings would break caching or isolation ([`deploy/preflight.md`](https://github.com/mirastacklabs-ai/miracache/blob/main/deploy/preflight.md)): ```sh PYTHONPATH=python python -m miracache.preflight --engine vllm --engine-args "" \ --secret-file /etc/miracache/keys/secret ```
vLLM: the offloading connector with miracache as a secondary tier ```sh vllm serve --block-size 16 --prefix-caching-hash-algo sha256_cbor \ --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both", "kv_load_failure_policy":"recompute", "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":34359738368, "canonical_layout":true, "secondary_tiers":[{"type":"MiraTier","module_path":"miracache.vllm_tier", "config":"/etc/miracache/client.toml","kv_load_failure_policy":"recompute"}]}}' ``` `kv_load_failure_policy` must be `recompute` (the default `fail` would turn a cache miss into a request error). Every host must use the same `PYTHONHASHSEED`, block size and hash algorithm; pipeline and context parallelism must be 1. If you pass `--revision` or `--quantization`, mirror them in the `MiraTier` entry as `"model_revision"` and `"ns_fields": {"quantization": "..."}`, so hosts serving different weights never share blocks (preflight checks it).
SGLang: HiCache with the dynamic storage backend ```sh python -m sglang.launch_server --model-path --page-size 64 \ --enable-hierarchical-cache \ --hicache-storage-backend dynamic \ --hicache-storage-backend-extra-config "$(cat deploy/sglang/extra-config.example.json)" ``` See [`deploy/sglang/README.md`](https://github.com/mirastacklabs-ai/miracache/blob/main/deploy/sglang/README.md) for every required setting.
**6. Watch it.** Each memory node serves Prometheus metrics on its `[telemetry] listen` address. Import the dashboards and alert rules from [`deploy/observability/`](https://github.com/mirastacklabs-ai/miracache/blob/main/deploy/observability/README.md). --- URL: https://www.mirastacklabs.ai/miracache/faq.md # miracache: frequently asked questions > Source: the FAQ on https://www.mirastacklabs.ai/miracache/ (generated at build time, 2026-11-19). The repo has no docs/faq.md yet; this file switches to it automatically when it appears. ## Do we need to change vLLM or SGLang? No. Zero changes to either engine: both load miracache as a plug-in through configuration flags only. ## Do we need a lossless (PFC) network? No. iWARP runs on plain Ethernet. ConnectX-6 Dx and newer are profiled for Zero-Touch RoCE (not yet verified on hardware). If you control your switch, PFC with ECN is supported too. ## Can we use InfiniBand? Not yet. miracache runs on RoCE v2 and iWARP over Ethernet. ConnectX cards that also support Ethernet work when switched to Ethernet mode on an Ethernet switch. ## Which model will we run? The model is chosen by the GPUs you bring. Every on-versus-off comparison uses the same model and settings on both sides. ## Do the memory servers need GPUs or fast CPUs? No. They serve reads with no CPU work on the data. Memory capacity and the network card matter most. ## Why not cache on local NVMe instead? NVMe is cheaper per TB but slower per read, wears out under constant rewrites, and only its own server can use it. The shared DRAM pool is faster, serves every GPU server and keeps a session’s state when it moves. NVMe works well as a lower tier for cold sessions. ## How do I install it? Start with the [README on GitHub](https://github.com/mirastacklabs-ai/miracache#readme). ## What if our servers are PCIe Gen3? They are welcome. A Gen3 x16 slot carries one 100G port at full speed.