# elelem-ggml

> A Qwen2 forward pass built by hand on ggml, with the tokenizer in Ruby. 466 lines of C and about 300 lines of Ruby from prompt to tensor ops.

Maturity: lab.  Source: https://src.mokhan.ca/elelem/ggml/


> **Experimental:** CPU only, greedy decoding only, no tool calls, and
> only Qwen2-family models decode correctly. Not published to RubyGems.

`elelem-ggml` runs a language model on [ggml](https://github.com/ggml-org/ggml),
the tensor library under llama.cpp, without llama.cpp. The transformer
graph is written by hand in one C file. The tokenizer and chat template
are plain Ruby. Use it to learn what happens between your prompt and the
next token. For real work, use [`elelem-llama`](/plugins/llama/).

## Install

The gem is not on RubyGems. Build it from source:

```bash
git clone https://git.mokhan.ca/elelem/ggml.git
cd ggml
git submodule update --init --recursive
bundle install
bundle exec rake compile
```

ggml is vendored as a git submodule at `vendor/ggml`. `rake compile` runs
`ext/elelem/ggml/extconf.rb`, which builds ggml with CMake, then compiles
`ext/elelem/ggml/elelem.c` into `ext/elelem/ggml/native/libelelem_ggml.so`
next to ggml's shared libraries.

## Configure

Point it at a GGUF file:

```bash
export ELELEM_GGUF_MODEL=/path/to/qwen2-model.gguf
```

Or put the file in `~/.agents/models/`. If that directory holds more than
one `.gguf`, symlink the one you want to `~/.agents/models/default.gguf`.
These are the same variable and directory `elelem-llama` uses.

The context size is the model's `context_length`, capped at 4096. The
plugin always uses 4 CPU threads. Neither is configurable from the
environment.

## Use

From the repository:

```bash
ruby -Ilib exe/elelem-ggml ask "hi"
```

`exe/elelem-ggml` is the elelem CLI with `ggml` as the default provider,
so `chat` (the default command) and `ask` both work. To use it from the
main `elelem` command, put `lib` on Ruby's load path:

```bash
RUBYLIB=/path/to/ggml/lib elelem -p ggml
```

The reply appears all at once when generation finishes. There is no
token streaming.

## How it works

A reply takes 5 steps. Each one is a short file you can read.

**1. Read the GGUF metadata in Ruby.** `lib/elelem/ggml/gguf.rb` parses the
GGUF header and key-value section with `IO#read` and `unpack`. It skips the
tensor data. The tokenizer needs 4 keys from it: `tokenizer.ggml.tokens`,
`tokenizer.ggml.merges`, `tokenizer.ggml.token_type`, and
`tokenizer.ggml.eos_token_id`.

**2. Format the prompt as ChatML.** `lib/elelem/ggml/chat_ml.rb` wraps each
message in `<|im_start|>role` and `<|im_end|>`, then ends with
`<|im_start|>assistant` so the model writes the next turn. If there is no
system message, it adds Qwen's default one. elelem always sends its own
system prompt, so that default only matters when you call the provider
directly.

**3. Tokenize with byte-level BPE.** ggml has no tokenizer, so
`lib/elelem/ggml/tokenizer.rb` implements one. Strings like `<|im_end|>`
that match a control token (`token_type` 3) map straight to their ID.
Everything else is split with Qwen2's pre-tokenizer regex, mapped
byte-by-byte to printable characters (the GPT-2 byte encoder), and merged
pair by pair, always taking the lowest-ranked merge, until no merge applies.
The specs pin the exact token IDs against a real Qwen2 vocabulary.

**4. Run the transformer in C.** `el_open` in `ext/elelem/ggml/elelem.c`
loads the GGUF with ggml's `gguf.h`, reads the hyperparameters under the
architecture's prefix (`qwen2.block_count`, `qwen2.embedding_length`, and
so on), and allocates the KV cache. Each forward pass builds a fresh ggml
graph. For every layer:

- **RMSNorm.** Scale the hidden state by its root mean square, then by a
  learned weight. No mean subtraction, no bias.
- **Attention projections.** Multiply by the Q, K, and V weights. Add the
  bias if the file has one.
- **RoPE, NEOX style.** Rotate Q and K by an angle that depends on token
  position, so attention scores encode relative distance. NEOX pairs
  dimension `i` with `i + d/2`, not with its neighbour.
- **Grouped-query attention (GQA).** There are fewer K/V heads than query
  heads. The test model has 14 query heads and 2 K/V heads. The cache only
  stores the 2, and ggml's matrix multiply broadcasts each one across 7
  query heads.
- **KV cache.** The new K and V are copied into the cache at the current
  position. Attention then reads every cached position, with a causal mask
  of `-INFINITY` above the diagonal and softmax scaled by
  `1/sqrt(head_dim)`.
- **Output projection and residual.** Project back to the model width and
  add the layer input.
- **SwiGLU feed-forward.** RMSNorm, then `down(silu(gate(x)) * up(x))`,
  then another residual add.

After the last layer comes a final RMSNorm and the output projection. If
the file has no `output.weight`, it reuses the token embedding (tied
weights).

**5. Pick tokens and decode.** Generation is greedy: C scans the logits for
the largest value. The first pass (prefill) runs the whole prompt at once
but projects only the last position to logits. Each later pass feeds one
token and reuses the cached K and V. It stops at the EOS token or
`ELELEM_GGUF_MAX_TOKENS`. Ruby turns the IDs back into bytes with the
inverse byte map and returns UTF-8 text.

Ruby talks to C through Fiddle with 4 functions: `el_open`, `el_version`,
`el_generate`, and `el_close`. Token IDs cross the boundary as packed
`int32` arrays.

The KV cache is F32, `head_dim × n_head_kv × n_ctx` per layer, for K and V.
It lives as long as the provider, but every reply starts again at position
0 and re-reads the whole conversation.

## Limitations

- Only Qwen2-family models decode correctly. The loader accepts other dense
  architectures (attention bias is optional, the output layer can be tied)
  and Llama-family files load without crashing, but RoPE ordering depends
  on the architecture. Qwen2 needs NEOX, Llama needs NORMAL, and no GGUF key
  says which. The code hardcodes NEOX, so Llama-family output is wrong. A
  spec with a 2-layer Llama-architecture model keeps the wrong output as a
  regression canary.
- Greedy decoding only. No temperature, top-p, or seed.
- No tool calls. The tool list elelem passes in is ignored, so the agent
  cannot read or edit files with this provider.
- CPU only. `n_gpu_layers` is accepted and ignored.
- No streaming. The reply arrives in one piece.
- If the prompt plus `ELELEM_GGUF_MAX_TOKENS` is more than the context size,
  the call fails with `ggml: generation failed`. elelem's system prompt and
  any `AGENTS.md` count toward the prompt.
- Every turn re-processes the whole conversation. Long sessions get slower.

## Development

```bash
bin/setup
bin/compile
bin/test
bin/lint
```

`bin/setup` runs `bundle install` only; run
`git submodule update --init --recursive` first. The tokenizer and ChatML
specs use a vocabulary-only Qwen2 GGUF in `spec/fixtures/`. Specs that need
real weights are skipped unless you set `ELELEM_GGML_TEST_MODEL`:

```bash
ELELEM_GGML_TEST_MODEL=/path/to/qwen2-model.gguf bin/test
```

Those specs expect a model with 24 layers, `n_embd` 896, 14 heads, 2 K/V
heads, `n_ff` 4864, and a 151,936-token vocabulary, and a greedy reply to
"hi" that starts with "Hello".


## Environment

| Variable | Default | Description |
|---|---|---|
| `ELELEM_GGUF_MODEL` | none | Path to a .gguf file. If unset, uses ~/.agents/models/default.gguf, or the only .gguf in ~/.agents/models/. |
| `ELELEM_GGUF_MAX_TOKENS` | `256` | Maximum number of tokens to generate per reply. |

Canonical: https://elelem.sh/plugins/ggml/