# elelem-llama2

> Runs Andrej Karpathy's llama2.c inside elelem. One C file does the whole forward pass; a 161-line C++ shim streams tokens back to Ruby.

Maturity: lab.  Source: https://src.mokhan.ca/elelem/llama2/


> **Experimental:** a story generator, not a coding assistant. No chat
> template, no tool calls, and replies show up as thinking text, not as
> the agent's answer. Not published to RubyGems.

`elelem-llama2` runs [karpathy/llama2.c](https://github.com/karpathy/llama2.c)
inside the elelem process. llama2.c is a single C file, `run.c`, that
implements Llama 2 inference with no dependencies. This plugin compiles
that file as-is and calls its functions from a small C++ shim. Use it to
see the smallest complete inference loop elelem can host, on tiny models
like the TinyStories checkpoints.

## Install

The gem is not on RubyGems. Build it from source:

```bash
git clone https://git.mokhan.ca/elelem/llama2.git
cd llama2
git submodule update --init --recursive
bundle install
bundle exec rake compile
```

llama2.c is vendored as a git submodule at `vendor/llama2.c`. There is no
CMake. `ext/elelem/llama2/extconf.rb` runs `cc` on `run.c` and `c++` on the
shim, and writes `ext/elelem/llama2/native/libelelem_llama2.so`.

## Configure

You need a checkpoint in llama2.c's own `.bin` format, made by
`vendor/llama2.c/export.py` or downloaded. The upstream llama2.c README
shows how to download `stories15M.bin`, a small TinyStories model. The vendored `tokenizer.bin` is used by default.

```bash
export ELELEM_LLAMA2_CHECKPOINT=/path/to/stories15M.bin
```

`ELELEM_LLAMA2_STEPS` counts every position, prompt tokens included. Keep
it at or below the model's `seq_len`: the shim does not clamp it.

## Use

```bash
bundle exec exe/elelem-llama2 ask "Once upon a time"
```

`exe/elelem-llama2` is the elelem CLI with `llama2` as the default
provider. `chat` (the default command) gives you a REPL.

Tokens stream as they are sampled, but the plugin sends them as
`thinking` events, not `saying`. In `chat` you watch the story appear as
thinking text, and the recorded reply is empty. `ask` streams the text,
then exits with `elelem: no reply`.

## How it works

`run.c` has no header and its own `main()`. The build compiles it to an
object file and links it into the shim. The shim never calls `main()`; it
calls the building blocks directly. Because there is no `.h` to include,
`ext/elelem/llama2/elelem.cpp` copies the struct layouts and function
signatures from `run.c`. If upstream changes them, the shim has to change
too.

**Open.** `el_open` calls 3 functions from `run.c`:

- `build_transformer` loads the checkpoint and sets up the weights,
  activations, and KV cache.
- `build_tokenizer` loads `tokenizer.bin`.
- `build_sampler` stores temperature, top-p, and the RNG seed.

**Prompt.** llama2.c has no chat template. The Ruby provider joins the
content of every message with newlines into one string. That includes
elelem's system prompt and any `AGENTS.md`.

**Generate.** `el_generate` is a short loop over `run.c`'s functions that
hands each piece to a callback:

1. `encode` turns the prompt into token IDs, with a BOS token in front.
2. For each position, `forward` runs one token through the network and
   returns logits.
3. While still inside the prompt, the next token is the next prompt token
   (teacher forcing). After the prompt, `sample` picks one from the logits
   with temperature and top-p.
4. `decode` turns the token into a string, and the shim calls Ruby's
   callback with it.
5. It stops at the step limit, or when the model emits BOS (ID 1), which
   llama2.c uses to separate sequences.

Ruby supplies the callback as a `Fiddle::Closure::BlockCaller` and calls
`el_generate` with `need_gvl: true`, so the callback can run Ruby code
from inside the C loop. Each piece is forced to UTF-8 and scrubbed before it
reaches elelem.

Every call starts again at position 0, so the KV cache is not reused across
turns.

## Limitations

- Only llama2.c's `.bin` checkpoint format and the original Llama 2
  architecture. No GGUF.
- No chat template and no tool calls. The tool list is ignored.
- Replies arrive as `thinking` events. The agent's answer is always empty,
  so `ask` ends with `elelem: no reply`.
- The prompt is echoed back. The callback fires for every position,
  including the prompt tokens being fed in.
- The whole conversation, system prompt included, is one prompt. On a
  small model with a short `seq_len` this crowds out the reply.
- `ELELEM_LLAMA2_STEPS` is not clamped to `seq_len`.
- No quantized models. Upstream's `runq.c` is not wired in.
- The gemspec does not package `vendor/llama2.c/tokenizer.bin`, so the
  default tokenizer path only works from a source checkout.

## Development

```bash
bin/setup
bin/compile
bin/lint
```

`bin/setup` runs `bundle install` only; run
`git submodule update --init --recursive` first. There is no test suite.


## Environment

| Variable | Default | Description |
|---|---|---|
| `ELELEM_LLAMA2_CHECKPOINT` | none | Required. Path to a llama2.c .bin checkpoint. |
| `ELELEM_LLAMA2_TOKENIZER` | `vendor/llama2.c/tokenizer.bin` | Path to a llama2.c tokenizer.bin. The default is the one in the vendored submodule. |
| `ELELEM_LLAMA2_STEPS` | `0` | Total positions to run, prompt included. 0 means the model's seq_len. |
| `ELELEM_LLAMA2_TEMP` | `1.0` | Sampling temperature. |
| `ELELEM_LLAMA2_TOPP` | `0.9` | Top-p (nucleus) sampling threshold. |
| `ELELEM_LLAMA2_SEED` | `-1` | Random seed. Negative uses the current time in nanoseconds. |

Canonical: https://elelem.sh/plugins/llama2/