Most AI products are a thin wrapper around someone else's stack, running on someone else's silicon. SiloLM stands on the best open-source foundation there is — llama.cpp — and underneath it we build what almost nobody else does: our own accelerator, designed, verified and programmed in-house.
We didn't reinvent tokenizers, samplers or model formats. SiloLM is a native ggml backend inside a llama.cpp fork: the scheduler hands our accelerator the workload that dominates all LLM inference, and everything else stays on the CPU — exactly where it belongs.
Our runtime accepts every ggml weight format — Q4, Q5, Q8, K-quants, IQ, MXFP4 and more — handles broadcasting, permuted views and strided tensors, and falls back to CPU transparently when the accelerator isn't connected.
Designed in-house down to the lowest level of the hardware, and verified bit-for-bit against a mathematical reference model. The heaviest compute in inference runs on silicon that is entirely ours.
A proprietary on-chip memory and compression pipeline keeps model data close to the arithmetic and makes every byte of bandwidth do real work. How it does that is the part we don't publish.
A wire protocol and host runtime map workloads of any shape onto the hardware and integrate seamlessly with the engine above — invisible to the models running on top.
Joules per token drives every design decision, from how model data lives on the chip to a dataflow where every byte of bandwidth does real work.
V E R I F I E D O N H A R D W A R E
The SiloLM backend passes llama.cpp's own backend conformance suite on the physical hardware — across every weight type, shape, broadcast and edge case. You can verify us with the tools you already trust.
Thirty tokens per second is faster than people read — from a device drawing less power than a laptop charger, with the model living entirely in the accelerator's own memory. Load once, generate forever.
| Current generation | Next generation | |
|---|---|---|
| Compute core | first-generation core @ 27 MHz | 16× wider, @ 200+ MHz |
| Model memory | one weight tile at a time (on-chip) | 1 GB DDR3 — whole models stay resident |
| Host link | USB-serial, ~30 KB/s readout | PCIe 3.0 ×4, ~3.5 GB/s (~10,000×) |
| Sustained compute | ~0.1 GFLOPS | 40–64 GFLOPS |
| Decode, 135M-class model | minutes per token | ~30 tokens/second |
| Energy per token | link-dominated | ~0.25 J projected |
After that comes the road we designed for from day one: dedicated silicon. The core was architected from the start to make that jump — a first custom chip is within reach of a small company's budget, not a moonshot — and it buys another order of magnitude in energy per token.
T H E F U T U R E I S L O C A L
Next-generation figures are engineering projections derived from the validated current-generation architecture; current-generation results are measured on hardware.