LLM inference,
planned in Ruby

Three jobs in one list

Read the 18 items again with one question in mind, "where does this code live?", and they sort themselves.

Inside the engine. PagedAttention manages the key-value (KV) cache, the per-request memory a transformer keeps so it doesn't recompute earlier tokens, in fixed-size blocks the way an operating system pages memory. Its authors report 2–4× higher throughput than earlier systems at the same latency. Continuous batching, KV eviction, prefix sharing, speculative decoding and quantized kernels all live in the same place: the loop that schedules GPU work every few milliseconds. That code is in vLLM, SGLang, llama.cpp and TensorRT-LLM, written in Python, C++ and CUDA. Rewriting any of it in Ruby would be a hobby, not an optimization.

Around the engine. Choosing which model answers a request, capping how many tokens it may spend, queueing when the GPU is full, recording latency, tokens, errors and cost, and testing it all under load: that's an HTTP service with policy in it. FrugalGPT reports matching GPT-4's quality at up to 98% lower cost by cascading cheaper models first, and RouteLLM reports cutting cost by more than half in some cases with a learned router. Both ideas are application logic. Ruby does that well, and it's the part we'd own anyway.

On top of both. Cost per million tokens, cost per finished task, and when renting a GPU beats paying per token. That's arithmetic over measurements, and the measurements are the hard part.

All 18 items, with honest calls

"Ruby" means we'd write it. "Engine" means we run someone else's Python, C++ or CUDA and talk to it over HTTP. The laptop column is about this machine: an i9-9880H, 30 GB of RAM and a GTX 1650 Max-Q with 4 GB (compute capability 7.5, the Turing generation).

Ten of the 18 are things we'd write in Ruby or configure ourselves. The other eight are things to understand, measure and choose between, which is still real work, just not Ruby work.

What a GTX 1650 can and can't do

Every claim in this section comes from the projects' own documentation, checked today.

• vLLM asks for compute capability 7.5 or higher, and the GTX 1650 is exactly 7.5. Its quantization table marks AWQ, GPTQ, Marlin, INT8 W8A8, bitsandbytes and GGUF as supported on Turing, and FP8 W8A8 as Ada (8.9) and newer only. So vLLM can start here, but the card reports about 3.7 GB usable, and model weights plus a useful KV cache for many concurrent requests won't fit in that for anything bigger than a small model. It's a place to learn the command line, not to benchmark.
• SGLang now ships only CUDA 13 builds. Our driver (595.99) is new enough for that. Its default attention backend, FlashInfer, supports sm75 and up, which includes this card. The same memory limit applies.
• TensorRT-LLM's support matrix lists Blackwell, Hopper, Ada and Ampere. It says unlisted architectures get community support only, so Turing is out.
• llama.cpp's server has continuous batching (on by default), parallel request slots, a draft model for speculative decoding, 8- and 4-bit KV-cache types and a Prometheus metrics endpoint. It's the one engine that exercises most of the roadmap's ideas inside 4 GB, and Ollama already ships a copy of it on this machine.
• The GPU is shared. Ollama keeps qwen3:4b loaded for 24 hours after use, and that alone takes 2.7 GB. Any experiment has to unload it first, and a permanent llama-server would compete with Ollama for the same memory. That's a decision for Jay, covered in the plan.

Bottom line: the laptop is good for learning how serving behaves (batching, queueing, quantization levels, KV pressure, drafts) and for building and load-testing the Ruby layer. It isn't good for throughput numbers anyone should compare to a data-centre GPU. Those come from a rented 24–80 GB card, run for a few hours with a written method.

A historical result: batching on 4 GB

Evidence corrected October 22, 2026. These are the published October 20 observations, not experiments independently rerun for this update. The complete original Ruby clients have now been recovered and published in the reproducibility bundle, with an evidence manifest and offline checks. The 26 historical result rows are unchanged.

The reported model was Ollama's qwen3:4b, a 2.5 GB quantized GGUF, using llama.cpp's server with 1, 4 and 8 parallel slots and 1,024 tokens of context per slot. The recovered client repeats a sentence about KV caches four times, prefixes each llama request with its index, and requests 200 tokens at temperature 0 and seed 1, with EOS ignored and prompt caching off. It sends one warmup request before the measured bursts. The exact prompt and options are in the manifest; the first numbered GPU prompt was logged as 70 tokens, not the previously stated 80. Ollama uses a different request shape and does not explicitly disable prompt caching.

Like-for-like table: every row below uses 16 total concurrent requests and 3,200 generated tokens. Aggregate throughput includes queueing and HTTP wall time. The former table mixed request counts.

Eight slots delivered about 2.66× the aggregate throughput of one slot in this fixed workload. This does not mean faster isolated responses: the single-request rows took 6.50, 6.65 and 6.62 seconds at 1, 4 and 8 slots. In the 16-request rows, backend-reported per-request decode rates fell from 32.2–32.8 to 11.9 tokens/s. Those rates exclude full client queueing and are not time to first token. The shorter batch completion time reflects concurrent work.

The original report also listed GPU memory at 2.67, 3.11 and 3.68 GB. Those figures are not fields in the JSONL and were not remeasured here; they do not establish a general eight-conversation capacity limit. The CPU-labelled rows report roughly 10 tokens/s at one slot and up to 22.4 in the measured four-slot workload. Ollama's measured aggregate stayed around 24–25.3 tokens/s, with 63.99 seconds for the slowest of eight requests. Its full historical service configuration is not preserved, so these observations do not diagnose its parallelism or establish an equal-cache comparison.

Limits: no repeated GPU trials, uncertainty estimate, output-quality evaluation, longer prompts, time-to-first-token measurement or load over 16 requests. The original scripts and explicit llama launch settings survived; the model blob identifier comes from historical commands and logs, not an independent rehash. Full inherited configuration and binary/library hashes remain missing. The bundle's checks establish artifact consistency, not reproduction of the experiment.

Download the original llama client, original Ollama client, manifest, checker, tests and offline fixture. The instructions explain safe checks and historical invocations. Running the original clients directly sends real inference requests, including a warmup even without arguments. No inference service was contacted or changed for this correction.

The Ruby part: one gateway

Items 6, 7, 10, 11, 12, 15, 17 and 18 collapse into one small service, which we'd call apps/gateway: an OpenAI-compatible HTTP endpoint (/v1/chat/completions, streaming included) that every app and agent on the machine calls instead of calling a provider directly. It follows rules we already have: one adapter per provider, config from the environment, OpenTelemetry out, registered in observability/services.yml.

What's in it.

• Adapters. One small class per backend: llama.cpp server, Ollama, OpenRouter, Anthropic, and "any OpenAI-compatible URL", which covers a rented vLLM or SGLang box with no new code. apps/intel already has the first version of this shape, an analyst with a local Ollama default and a swappable OpenRouter class; the gateway generalizes it so every app shares it.
• Router. Rules first, in a YAML file: route by task type, prompt length, required quality tier, tenant and current queue depth; fall back from local to hosted when the local queue is too long; never send a tenant's data to a provider that tenant hasn't allowed. FrugalGPT's cascade (try cheap, escalate if a check fails) is the natural second step. RouteLLM's learned router is a third step only if the rules measurably miss.
• Budgets. Every request carries a tenant and a task. The gateway counts prompt tokens before sending (the tokenizers gem reads Hugging Face tokenizer files), sets max_tokens from what's left in that tenant's budget, refuses or downgrades when the budget is gone, and records actual use afterwards. This is the same token frugality apps/intel already practises: collect and rank without a model, and only let a model read the short list.
• Admission and queueing. The engine does continuous batching; the gateway decides who gets in. A bounded queue per backend, priorities per task, and a fast "busy, try the fallback" answer instead of a request that waits 105 seconds, which is what one slot did to the slowest of sixteen requests above.
• Telemetry. A span per request with model, provider, tokens in and out, time to first token, time per output token, queue time, outcome and cost, sent to the local collector. Prometheus also scrapes llama-server's /metrics (and vLLM's, when we have it), so one Grafana dashboard shows the request, the queue and the GPU together. The gpu-metrics exporter already reports the card.

The simpler option, stated first. LiteLLM already does most of this: one OpenAI-format gateway over 100+ providers, with per-project and per-user spend tracking, routing with fallbacks, and load balancing. It's Python and open source, and we could run it as a backing service today. We'd still write our own small Ruby gateway, for three reasons: the routing and budget rules are our policy and belong in code we test; tenants and grants come from our own database, per our identity rules; and we need four providers, not a hundred. If the Ruby version passes 500 lines, or we find ourselves rebuilding provider quirks, we should stop and run LiteLLM instead.

Ruby pieces that already exist. All checked on rubygems.org today: ruby_llm 2.1.1 for calling hosted providers from apps, async-http 0.105 for a streaming proxy and a load generator that can hold 1,000 open connections in one process, opentelemetry-sdk 1.13 for spans, tokenizers 0.7 for counting, llama_cpp 0.28 if we ever want llama.cpp in-process (we don't; HTTP keeps the engine swappable), and onnxruntime 0.11 for small classifier models, such as a learned router. ONNX Runtime GenAI, the generation loop, has Python, C#, C/C++ and Java APIs but no Ruby one, so edge generation stays in other languages.

The plan, step by step

Each step has a check that says it's done. Steps 1–4 run on the laptop at no cost. Step 5 needs money and a provider, so it's Jay's call.

Step 0: historical baseline (reproduction incomplete)

Raw observations and recovered clients are published above. Offline consistency checks do not establish an independently reproduced baseline; missing historical configuration remains explicit in the manifest.

Step 1: the gateway, minimum version

Roda on Falcon, one endpoint, three adapters (llama-server, Ollama, OpenRouter), rule-based routing, per-tenant token budgets, OpenTelemetry spans. Minitest first, with expected results worked out by hand: a request over budget is refused or downgraded and never reaches a provider; a tenant restricted to local models is never sent to OpenRouter; a stream that fails half-way is recorded as an error with the tokens it actually used. Check: tests pass, the service is in services.yml and healthy, and one forced error shows up in Grafana.

Step 2: a llama-server service and the engine experiments

Run llama.cpp's server as a user systemd service with --metrics. Because Ollama and llama-server can't both hold a 4B model in 4 GB, pick one of two set-ups: llama-server replaces Ollama for text (simplest, and it batches), or Ollama stays and llama-server only runs during experiments. Then run the engine experiments one at a time, each a single changed setting against the step 0 baseline: KV cache at 8 and 4 bits, GGUF quantization levels from 8-bit down to 4-bit (with a small fixed set of our own eval questions, since speed without quality is meaningless), and a small draft model for speculative decoding with its acceptance rate. Check: each experiment has a row of numbers and a sentence on what changed.

Step 3: load tests

A Ruby load generator on async-http that holds N concurrent streams and records time to first token and total time per request. Two separate tests, never mixed: the gateway at 1,000+ concurrent against a fake backend that streams canned tokens (this measures our own overhead and queueing, the only thing a laptop can honestly test at that scale), and the real model at 1–16 concurrent (this finds the saturation curve, already sketched above). Check: we can say "the gateway adds X ms at the median and Y at the 99th percentile at 1,000 concurrent" with the raw data published.

Step 4: the dashboard and one public service

An "Inference" Grafana dashboard: requests and errors by provider, time to first token, tokens per second, queue depth, KV cache use, GPU memory and utilization, and spend against budget. Then expose one endpoint publicly behind an API key, such as a small summarizer that routes local-first and only escalates on demand, with its benchmark page. Check: the service is healthy in the dashboard, and an error from it can be found there.

Step 5: a rented-GPU benchmark day (needs Jay)

One rented 24–80 GB GPU for a few hours. Same model, same prompts, same load script: llama.cpp, vLLM and SGLang; FP16 against AWQ, GPTQ and FP8; with and without speculative decoding. Point the gateway's "OpenAI-compatible URL" adapter at it, so the Ruby layer is tested against real engines too. Write the method before renting, so the money buys answers to questions we already wrote down. Check: a public page with the method, versions, hardware and raw data, and a plain list of what we didn't test.

What we'd skip

Kubernetes autoscaling, until there's more than one machine and a measured reason; one laptop with systemd is the right size for this. TensorRT-LLM on this card. Writing kernels, a Ruby PagedAttention, or a Ruby inference engine. And chasing a 1,000-user number on hardware tested only up to eight server slots, because a benchmark that can't be reproduced on comparable hardware teaches the reader nothing.

Unit economics, without invented prices

The one formula that matters for self-hosting:

cost per million output tokens = hourly cost ÷ (tokens per second × 3,600) × 1,000,000

The fixed-16 historical rows give the denominators: 30.3 tokens/s is 109,080 tokens/hour and 80.7 is 290,520. At a hypothetical all-in cost of $1.00/hour, those are about $9.17 and $3.44 per million output tokens, a 62.5% difference. This is an illustrative calculation from one workload, not measured operating cost or a promise of lower cost per completed task.

Three cautions. The laptop's real hourly cost (electricity, wear, the fact that it's busy doing other things) isn't measured, so we haven't put a number on it. Per-token cost is the wrong unit for agents: what matters is cost per finished task, which includes retries, tool calls and how many tokens a weaker model wastes getting there, so the gateway should record cost per task as well as per request. And hosted per-token prices change often, so the gateway should read them from config and the dashboard should show what we actually paid, not what a price list said once.

Right now the OpenRouter balance is $0, so the default route for everything is the local model. That's a fine place to start, because it forces the router and budgets to earn any money they eventually spend.

What to read instead of release news

All are primary papers, each checked on arXiv today. The figures in brackets are the authors' own, measured on their hardware; they're reasons to read, not numbers to expect here.

• Memory and scheduling. PagedAttention / vLLM, Kwon et al. 2023 (2–4× throughput at equal latency). SGLang and RadixAttention, Zheng et al. 2023 (up to 6.4× on structured workloads). Sarathi-Serve, Agrawal et al. 2024, on the throughput-latency trade in batching. DistServe, Zhong et al. 2024, splitting prefill from decoding. FlashAttention, Dao et al. 2022, the IO-aware kernel underneath much of this.
• KV cache. H2O, Zhang et al. 2023, keeping only "heavy hitter" tokens. StreamingLLM, Xiao et al. 2023, attention sinks for unbounded streams.
• Speculative decoding. Leviathan et al. 2022 and Chen et al. 2023, the two original papers (2–3× with identical outputs). EAGLE, Li et al. 2024, drafting at the feature level.
• Quantization. LLM.int8(), Dettmers et al. 2022. GPTQ, Frantar et al. 2022. SmoothQuant, Xiao et al. 2022 (W8A8). AWQ, Lin et al. 2023 (protecting about 1% of salient weights). FP8 formats, Micikevicius et al. 2022.
• Routing and cost. FrugalGPT, Chen et al. 2023. RouteLLM, Ong et al. 2024.

A useful habit: when a release note claims a speed-up, find which of these ideas it implements, and on what hardware and batch size it was measured. Most claims become much less surprising.

What we won't claim

• That anything on this laptop predicts data-centre performance. The benchmark above is about this card's behaviour, not about qwen3:4b or llama.cpp in general.
• That Ruby will be in the hot path of inference. It won't, and it shouldn't. Ruby's job is policy, accounting and glue, where it's very good.
• Any figure from a paper as a figure for us. Speed-ups depend on model, hardware, batch size and workload.
• That the gateway is fast enough. That's what step 3 measures. Until then it's an assumption.

How this page was made: papers and docs were pulled with our own collection tools and checked against the originals, robots.txt respected, no model-written numbers kept. The GPU benchmark was run once per setting on October 20, 2026 (Ollama concurrency 1 has two rows), with llama.cpp's server as bundled in Ollama 0.40.1 (its build reports version 0.5.0-dev, commit 631109b34) and the NVIDIA 595.99 driver. Single runs carry noise; without repeated trials its magnitude is unknown. On October 22 we recovered the original clients and explicit launch arguments, corrected the comparison and ran offline consistency checks; we did not rerun inference.

Sources

1. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023.
2. Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, 2023.
3. Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, 2024.
4. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, 2024.
5. Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022.
6. Zhang et al., H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, 2023.
7. Xiao et al., Efficient Streaming Language Models with Attention Sinks, 2023.
8. Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022.
9. Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling, 2023.
10. Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, 2024.
11. Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, 2022.
12. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022.
13. Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, 2022.
14. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, 2023.
15. Micikevicius et al., FP8 Formats for Deep Learning, 2022.
16. Chen, Zaharia and Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, 2023.
17. Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, 2024.
18. vLLM docs: GPU installation requirements, quantization and supported hardware, metrics.
19. SGLang docs: installation (CUDA 13, FlashInfer sm75+).
20. NVIDIA: TensorRT-LLM support matrix.
21. llama.cpp: server README (parallel slots, continuous batching, draft models, KV types, metrics).
22. WebLLM (in-browser, WebGPU, OpenAI-compatible), ONNX Runtime GenAI (API languages and execution providers), LiteLLM (gateway, spend tracking, routing).
23. rubygems.org pages for ruby_llm, async-http, opentelemetry-sdk, tokenizers, llama_cpp and onnxruntime, versions as of October 20, 2026.
24. Our raw benchmark data: inference-roadmap-bench.jsonl.