The LLM
Neuron LLM
A language model whose weights live in a Solana account and whose every token is computed by a Solana program (neuron_llm). The math is integer-only, so every validator that runs it gets the same answer, bit for bit.
On Solana devnet. The model is locked in a devnet account and the program has written a first story there, identical to the off-chain run (details). Mainnet is next.
Measured
From rollup/sequencer/tests/llm.rs, generating from an empty prompt with greedy decoding. In every run the on-chain tokens are identical to the off-chain integer reference, and a lockstep test compares the whole session account with the host after every layer.
| Build | Story | Compute per token | Transactions per token |
|---|---|---|---|
| Plain bytecode (runs on stock Solana) | 127 tokens | 2.09M CU | 1.91 |
sol_infer (Neuron rollup) | 255 tokens | 0.36M CU | 0.27 |
The build deployed on devnet can't read its remaining compute (that syscall is off on devnet and mainnet), so it budgets each step with a conservative estimate. Measured over a full 255-token story, it takes 2.85 transactions per token: the figure on the home page.
Each transaction uses up to 1.4M compute units, Solana's per-transaction limit. Uploading the 313,751-byte model takes 327 transactions, each under the 1,232-byte packet limit. The story it writes from nothing begins: "Once upon a time, there was a little girl named Lily. She loved to play outside in the park."
Speed
A Step transaction doesn't name a position: it advances the session from wherever it is. So clients can send many at once and the chain runs them back to back, several in one block. npm run llm:live does exactly that against a running neuron-node: it uploads the model, starts a story, sends a full block of Step transactions every block and prints tokens as they land.
| Run | Story | Blocks | Speed |
|---|---|---|---|
| Live node over RPC, 34 Step transactions per block, sampled | 251 tokens | 3 | ~230 tokens/s |
| Live node over RPC, 34 per block, greedy (byte-identical to the host reference) | 251 tokens | 2 | ~370 tokens/s |
Sequencer harness, 8 per block (llm_many_steps_per_block) | 255 tokens | 9 | 30 to 31 tokens per block |
A full 48M CU block holds about 125 tokens. The sequencer executes about 525 tokens per second of CPU time on an Apple M4 Pro, where the SVM runs interpreted (Agave's JIT is x86-only). On Solana mainnet, a session is a single writable account, and Agave caps those at 12M compute units per block: about 5 tokens per block (~12 tokens/s) for one story, if a leader packs it fully. Separate sessions run in parallel.
Where the compute goes, per token, from the profile build (llm_profile): early in a story the weight matmuls are about 75% of the bytecode build's cost; late in a story attention catches up, because it grows with every token. The 2026-11-21 pass rebuilt attention (transposed key cache, flash-style value sum in 64-bit, scales folded at write time) and cut a token at position 200 from 5.26M to 2.84M CU in bytecode. On the rollup, attention now runs natively through sol_infer_attn_i8, so a layer costs a flat ~66K CU at any position and a token about 355K.
On Solana devnet
The plain-bytecode build is deployed on devnet at BfH35VV5cSVsxqU16YJLJQSvSKQjygY5X12Hn8Me5yMt. The model is locked in CutfHeMx…Js7S (SHA-256 9a1efbc0…e439c), and session 4pRrT19F…SUK1 wrote 123 tokens, identical to the host's integer reference. That run was throttled for the public devnet RPC (about one request a second); with a private endpoint a story runs at the account limit, about 4 tokens per block.
On a real Solana validator
The plain-bytecode build deployed to solana-test-validator (stock Agave 3.0.13: no rollup, no custom syscalls) at BfH35VV5cSVsxqU16YJLJQSvSKQjygY5X12Hn8Me5yMt, the address it will have on devnet and mainnet. The model uploaded and locked in 329 transactions (6 s), and a greedy session wrote 123 tokens in 14 s, about 4 tokens per block, identical to the host's integer reference. Four tokens per block is the limit Agave sets for one writable account (12M compute units per block).
sh llm/deploy.sh devnet # keys in ~/.config/neuron
node llm/run-chain.cjs --cluster devnet --greedy # upload once, then a verified story
The agent terminal
Neuron Terminal runs a cast of characters on the model. Each character is its own session account. A turn restarts the speaker's session with the story so far and a cue (Lily said to Ben, "), then streams Step transactions until the model closes the quote; every third turn the narrator continues the story instead. Every six turns, the words each character uses most join their persona line, which the model reads on their next turn. npm run terminal runs it live on a local rollup node with a feed the page connects to; elsewhere the page replays a recorded live session.
The model
v0 is stories260K from Andrej Karpathy's llama2.c (MIT license), trained on the TinyStories dataset. It's a small model with the same architecture as Llama: RMSNorm, rotary position embeddings, grouped-query attention and a SwiGLU feed-forward.
| Weights | 260K (dim 64, 5 layers, 8 heads sharing 4 key/value heads, feed-forward 172) |
| Vocabulary | 512 tokens, which keeps the classifier small |
| Context | Up to 512 tokens, set per session |
| On chain | 313,751 bytes: int8 weights plus scales, RoPE tables and the vocabulary (the feed-forward is padded from 172 to 176 with zero weights, which computes the same function and keeps every matmul in 8-column blocks) |
llm/fetch.sh downloads the checkpoint and checks its SHA-256. The host tools reproduce llama2.c's reference output word for word in float, then convert it to the on-chain format.
Integer-only inference
Solana has no floating point, and two validators must never disagree. Every step uses whole numbers, in llm/core, a no_std crate compiled into both the program and the host tools:
- Weights are int8, with one scale per output column. Activations are 16.16 fixed point and are quantized to int8 before each matrix multiply.
- RMSNorm uses an integer square root. Softmax and SiLU use a fixed-point
2xlookup table. RoPE reads precomputed tables stored in the model account. - The key/value cache is int8 with one scale per head vector, stored in the session account.
- Sampling is greedy, or seeded: softmax at a temperature with a min-p cutoff, drawn by a xorshift generator whose seed is stored in the session, so anyone can replay a story.
The integer path writes the same text as the float model: identical for 71 of the first 255 tokens of the reference story, and for all 118 tokens of another prompt. After that, small rounding differences change a few words, and the story stays coherent.
Two builds
- Default (plain bytecode): matrix multiplies use a packed kernel. Values are biased to unsigned so each 64-bit register holds two 32-bit lanes, and one multiply instruction does two multiply-adds. This is the build that runs on stock Solana, mainnet included.
sol-infer: matrix multiplies callsol_infer_matmul_i8, and each attention head callssol_infer_attn_i8, which runs the very same core function (attend_head) natively. Both syscalls only exist on the Neuron rollup.
All three kernels (reference, packed, syscall) give bit-identical results; the tests check it.
Program
Accounts
- Model (owned by the program): a 96-byte header (magic
NMDL, state, blob length, authority, SHA-256), then the model blob. - Session (owned by the program): the session's authority (32 bytes), then position, tokens, sampling settings, the residual vector between layers, and the int8 key/value cache (keys stored transposed). A 256-token session is 144,256 bytes.
Instructions
| Tag | Instruction | What it does |
|---|---|---|
| 0 | InitModel | Claims a fresh model account and records its authority. |
| 1 | WriteModel | Writes a chunk of the blob. Authority only, while loading. |
| 2 | FinalizeModel | Validates the blob, stores its SHA-256, and locks it for good. |
| 3 | StartSession | Starts a generation from a prompt (BOS first), with greedy or seeded sampling. A restart needs the session's authority. |
| 4 | Step | Runs layers until the story ends, the unit limit is hit, or the compute left can't fit the costliest unit so far. Anyone can call it. Each generated token is written to the log. |
A token is six units: five layers, then the classifier. They can span transactions, so a token too big for one transaction still completes.
Run it
npm run llm:fetch # weights (sha256-checked)
npm run llm -- float # float reference: llama2.c's output, word for word
npm run llm -- int --kernel packed --temp 0.8 --min-p 0.1 --seed 7
npm run llm:build-programs # neuron_llm.so and neuron_llm_rollup.so
npm run llm:test # host tests, then the model on the real SVM
Mainnet costs (estimate)
From the measured transaction counts and Solana's current rent and fee rules:
| Model account (rent deposit) | ~2.19 SOL |
| Uploading the weights | 327 transactions, ~0.0016 SOL |
| A 256-token session (rent deposit, reusable) | ~1.0 SOL |
| Deploying the program (101 KB) | ~0.7 to 1.4 SOL, depending on room left for upgrades |
| Per token | 1.9 transactions: ~9,500 lamports in base fees, plus priority fees |
The program has no close instruction yet, so model and session deposits stay locked until one is added.
Next
- Deploy to mainnet, then make the program immutable. (Devnet is done: a first story, verified token for token.)
- Make attention cheaper. It grows with story length and is now the largest per-token cost.
- Train our own model (custom vocabulary, fewer bits per weight), then a 1B-class model on the rollup.