# elelem-candle

> A skeleton provider that calls Rust from Ruby through rb_sys and magnus. The FFI and token streaming work; the model is a stub that echoes your prompt.

Maturity: lab.  Source: https://src.mokhan.ca/elelem/candle/


> **Experimental:** this is a skeleton. There is no inference.
> `Model#generate` echoes the prompt back word by word. Not published to
> RubyGems.

`elelem-candle` is the start of a provider backed by
[huggingface/candle](https://github.com/huggingface/candle), a tensor and ML
framework written in Rust. What works today is the boundary: Ruby calls a
Rust method in the same process, Rust calls a Ruby block once per token,
and Rust errors come back as Ruby exceptions. Read it to see how a Rust
inference engine plugs into elelem without a subprocess.

## Install

The gem is not on RubyGems. Build it from source:

```bash
git clone https://git.mokhan.ca/elelem/candle.git
cd candle
bundle install
bundle exec rake compile
```

There are no submodules. Cargo fetches candle and the other crates. The
build compiles `candle-core`, `candle-nn`, `candle-transformers`,
`tokenizers`, and `hf-hub` even though the stub does not use them yet, so
the first build takes a while.

## Configure

```bash
export ELELEM_CANDLE_MODEL=/path/to/model
```

The provider raises if `ELELEM_CANDLE_MODEL` is unset. Any non-empty value
works, because nothing is read from it yet.

## Use

```bash
bundle exec exe/elelem-candle ask "hi"
```

The reply is the stub's echo: `[stub:MODEL on DEVICE]`, followed by every
message in the conversation as `role: content`, elelem's system prompt
included. Each word arrives as a separate `thinking` event, then the whole
text arrives once as the answer.

## How it works

There are 3 layers, each short.

**Ruby provider** (`lib/elelem/candle/provider.rb`, 33 lines). `fetch`
flattens the messages into `role: content` lines, builds a `Proc` that
forwards each piece to elelem as a `thinking` event and appends it to a
buffer, and calls `Elelem::Candle::Model#generate(prompt, token_cb)`. When
it returns, the buffer goes to elelem as a `saying` event.

**Rust extension** (`ext/elelem/candle/src/lib.rs`, 75 lines). The
`#[magnus::init]` function defines `Elelem::Candle::Model` with `new` and
`generate`. `#[magnus::wrap]` makes a Rust struct a Ruby object, so
`Model.new(path, device)` returns a Ruby object that owns Rust memory.
`generate` takes the prompt as a `String` and the callback as a magnus
`Proc`, and calls `token_cb.call` for each piece. If the model path is
empty, it raises a `RuntimeError` in Ruby. The callback is an explicit
argument, not a block, because magnus converts positional arguments, not
an implicit block.

**Build.** [rb_sys](https://github.com/oxidize-rb/rb_sys) drives Cargo from
Ruby's extension machinery. `extconf.rb` is one call,
`create_rust_makefile("elelem/candle/candle")`. The Rakefile's
`RbSys::ExtensionTask` puts the compiled library in `lib/elelem/candle/`,
and `lib/elelem/candle.rb` loads it with `require "elelem/candle/candle"`.
This is the layout `bundle gem --ext=rust` generates.

Compare [`elelem-ratatui`](/plugins/ratatui/), which runs a separate Rust
binary and talks JSON over stdio. Here Rust is linked into the Ruby
process, with typed arguments and no serialization.

The source lists what comes next: load a tokenizer, load safetensors
weights onto a candle device, build a model from `candle-transformers`,
and replace the stub with a real forward pass and sampling loop.

## Limitations

- No inference. The output is the prompt echoed back.
- No chat template and no tool calls. The tool list is ignored.
- `ELELEM_CANDLE_MODEL` and `ELELEM_CANDLE_DEVICE` are only stored and
  printed.
- No GPU. The Cargo manifest notes the `cuda` and `metal` features are not
  enabled yet.
- No test suite.

## Development

```bash
bin/setup
bin/compile
bin/lint
```

`bin/compile` runs `bundle exec rake compile`. `bundle exec rake build`
compiles first, then builds the gem.


## Environment

| Variable | Default | Description |
|---|---|---|
| `ELELEM_CANDLE_MODEL` | none | Required. A model path or Hugging Face repo ID. Recorded and echoed back; nothing is loaded yet. |
| `ELELEM_CANDLE_DEVICE` | `cpu` | Device name. Recorded and echoed back; not used yet. |

Canonical: https://elelem.sh/plugins/candle/