# elelem-diffusion

> Adds image, video, and upscale tools backed by stable-diffusion.cpp, compiled into the elelem process and called through Fiddle.

Maturity: lab.  Source: https://src.mokhan.ca/elelem/diffusion/


> **Experimental:** no README, no published gem, and the specs mock the
> native library, so they do not exercise a real model.

`elelem-diffusion` gives the agent 3 tools: `image` (text to image, or
image to image), `video` (text to PNG frames), and `upscale`. They run
[stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp)
inside the elelem process, on your machine. Any provider that supports
tool calls can use them.

## Install

The gem is not on RubyGems. Build it from source:

```bash
git clone https://git.mokhan.ca/elelem/diffusion.git
cd diffusion
git submodule update --init --recursive
bundle install
bundle exec rake compile
```

stable-diffusion.cpp is vendored as a git submodule at
`vendor/stable-diffusion.cpp`, with its own ggml submodule inside, hence
`--recursive`. The build writes
`ext/elelem/diffusion/native/libelelem_diffusion.so` (`.dylib` on macOS).

This is a tool plugin, not a provider, so elelem does not load it on its
own. Add a plugin file that puts it on the load path:

```ruby
# ~/.agents/plugins/diffusion.rb
$LOAD_PATH.unshift(File.expand_path("~/src/diffusion/lib"))
require "elelem/diffusion"
```

elelem loads every `.rb` file in `~/.agents/plugins/` and
`.agents/plugins/` at startup. Adjust the path to where you cloned the
repository.

## Configure

Each tool reads its own variables when it is called. A missing variable
does not stop elelem from starting. The tool returns an error to the model
instead, for example
`{"ok":false,"error":"ELELEM_SD_IMAGE_MODEL is not set"}`.

| Tool | Needs |
|---|---|
| `image` | `ELELEM_SD_IMAGE_MODEL` |
| `video` | `ELELEM_SD_VIDEO_MODEL`, `ELELEM_SD_VIDEO_VAE`, `ELELEM_SD_VIDEO_T5XXL` |
| `upscale` | `ELELEM_SD_ESRGAN_MODEL` |

```bash
export ELELEM_SD_IMAGE_MODEL=/path/to/sd-model.gguf
export ELELEM_SD_ESRGAN_MODEL=/path/to/esrgan.pth
```

If the native library was not built, the plugin prints a warning and adds
no tools.

## Use

Start elelem with any provider and ask for an image:

```bash
elelem chat
```

```text
> Draw a lighthouse at dusk and save it to lighthouse.png, then upscale it to lighthouse-4x.png
```

The model decides when to call the tools. Their parameters:

| Tool | Required | Optional |
|---|---|---|
| `image` | `prompt`, `path` | `input_path`, `strength` (0.0 to 1.0, default 0.5), `steps` (default 20), `width` and `height` (default 512) |
| `video` | `prompt`, `output_dir` | `video_frames` (default 16), `fps` (default 8), `steps` (default 20) |
| `upscale` | `input_path`, `path` | none |

`image` with `input_path` does img2img: it edits the input image, and
`strength` sets how much changes. Width and height then come from the
input image. `video` writes `frame-0000.png`, `frame-0001.png`, and so on
into `output_dir`, which it creates. `upscale` scales by whatever factor
the ESRGAN model was trained for. On success every tool returns
`{"ok":true,"path":...}`.

## How it works

**Build.** `ext/elelem/diffusion/extconf.rb` configures stable-diffusion.cpp
with CMake as a static library, with examples, WebP, and WebM turned off.
It then links `ext/elelem/diffusion/elelem.cpp` together with
`libstable-diffusion.a` and every ggml archive into one shared library.
The shim is compiled with hidden visibility and, on Linux, linked with
`--exclude-libs,ALL`, so only the 7 `el_sd_*` functions are exported. The
copy of ggml inside stays private to this library.

**Shim.** `elelem.cpp` is 217 lines. Each function fills in one of
stable-diffusion.cpp's parameter structs and calls one library function:

- `el_sd_open` creates a context from a single model file
  (`model_path`).
- `el_sd_open_video` creates a context from 3 files: the diffusion model,
  the VAE, and the T5-XXL text encoder.
- `el_sd_generate` calls `generate_image` with prompt, size, steps, and
  seed, and writes the first image as PNG.
- `el_sd_img2img` reads the input with stb_image as 3-channel RGB, sets it
  as `init_image` with `strength`, and calls `generate_image`.
- `el_sd_video` calls `generate_video` and writes each frame as a PNG.
  Any audio it returns is freed and dropped.
- `el_sd_upscale` creates an ESRGAN upscaler, asks it for its scale
  factor, upscales once, and frees it.

stable-diffusion.cpp's log and progress callbacks are replaced with no-ops,
so it does not write to your terminal.

**Ruby.** `lib/elelem/diffusion.rb` binds the functions with Fiddle and
registers the tools in a `config.setup(:diffusion)` block. Contexts are
cached for the life of the process: one
per image model path, one per set of video model files. The ESRGAN upscaler
is not cached. Every native call runs under a single mutex, so only one
generation runs at a time. The generate calls set `need_gvl: false`, so
Ruby releases its global lock while C works and other Ruby threads keep
running. Thread count is `Etc.nprocessors`.

## Limitations

- The seed is always 42 and the negative prompt is always empty. The same
  prompt and settings give the same image.
- `video` always renders at 512 by 512. Width and height are not exposed.
- Output is always PNG, whatever extension `path` has. Video comes out as
  frames, not a video file.
- Model contexts are never freed until elelem exits. An image model and a
  video model can both stay in memory.
- Thread count is always `Etc.nprocessors`. There is no setting for it.
- Calls are serialized. A second request waits for the first.
- The tool description calls video "slow: minutes per clip".
- The specs replace the native library with test doubles. They check the
  Ruby wiring and defaults, not image output.

## Development

```bash
bin/setup
bin/compile
bin/test
bin/lint
```

`bin/setup` runs `bundle install` only; run
`git submodule update --init --recursive` first. `bin/test` runs RSpec and
does not need the native library or any model files.


## Environment

| Variable | Default | Description |
|---|---|---|
| `ELELEM_SD_IMAGE_MODEL` | none | Model file for the image tool, passed to stable-diffusion.cpp as model_path. |
| `ELELEM_SD_VIDEO_MODEL` | none | Diffusion model file for the video tool (a Wan2.1 or Wan2.2 model). |
| `ELELEM_SD_VIDEO_VAE` | none | VAE file for the video tool. |
| `ELELEM_SD_VIDEO_T5XXL` | none | T5-XXL text encoder file for the video tool. |
| `ELELEM_SD_ESRGAN_MODEL` | none | ESRGAN model file for the upscale tool. |

Canonical: https://elelem.sh/plugins/diffusion/