elelem-llama

elelem-llama runs a GGUF model inside the elelem process. The gem builds llama.cpp into a shared library when you install it, and the llama provider calls it through Ruby’s Fiddle. There is no server to start and no network call: your code and prompts stay on your machine.

Install #

gem install elelem elelem-builtins elelem-llama

elelem-builtins is optional but gives the agent its default tools. Requires Ruby 4.0, a C++17 compiler, and CMake. The gem ships the llama.cpp source and compiles it during gem install.

The build picks a backend in this order:

To force one, set ELELEM_LLAMA_BACKEND at install time:

ELELEM_LLAMA_BACKEND=vulkan gem install elelem-llama

If a GPU build fails, the build falls back to CPU and says so. The cpu backend compiles for the host CPU’s instruction set.

Configure #

Put a model in ~/.agents/models:

mkdir -p ~/.agents/models
mv ~/Downloads/model.gguf ~/.agents/models/

The provider finds the model in this order:

1. --model PATH
2. ELELEM_GGUF_MODEL
3. ~/.agents/models/default.gguf, if it exists (a symlink is fine)
4. the only .gguf file in ~/.agents/models

With several models in that directory and no default.gguf, it stops and asks you to pick one:

ln -s ~/.agents/models/model.gguf ~/.agents/models/default.gguf

GPU offload. When ELELEM_GGUF_N_GPU_LAYERS is unset, the provider offloads all layers (999) if the build used cuda, vulkan, or metal (Apple Silicon only) and the model file is smaller than 80% of the GPU’s free memory. Otherwise it uses 0 and runs on the CPU. A hip build always defaults to 0; set ELELEM_GGUF_N_GPU_LAYERS=999 to use the GPU.

Context. With ELELEM_GGUF_N_CTX=0, the context is the model’s trained length, capped at 65,536 tokens. If that does not fit in memory, the provider halves it until it does, down to 4,096.

Sampling. Top-k, top-p, temperature, and the seeded sampler apply when ELELEM_GGUF_TEMP is above 0. At 0 or below, sampling is greedy and the other sampling variables are ignored.

Use #

elelem -p llama
elelem -p llama --model ~/.agents/models/other.gguf
ELELEM_GGUF_N_CTX=32768 elelem ask -p llama "list the TODOs in lib/"

--model takes a file path, not a model name.

The gem also installs elelem-llama, which is elelem with llama as the default provider:

elelem-llama chat

Unlike elelem, elelem-llama does not load elelem-builtins, so the agent has no tools unless a plugin file in ~/.agents/plugins adds them.

How it works #

gem install runs ext/elelem/llama/extconf.rb. It configures the vendored llama.cpp with CMake (shared libraries, no server, tools, examples, or tests), builds the llama and llama-common targets, then compiles elelem.cpp into a small shim, libelelem_llama.so (.dylib on macOS). The shim and the llama.cpp libraries are copied into the gem’s ext/elelem/llama/native directory, along with a file that records the backend.

At runtime the provider opens the shim with Fiddle and calls 4 C functions: el_open loads the model and creates a context, el_generate runs one turn, el_close frees it at exit, and el_gpu_free_memory reports free GPU memory for the offload default. The model loads once, when the provider is built.

Each turn, the shim formats elelem’s messages and tools with the model’s own chat template (Jinja), tokenizes the prompt, and reuses the KV cache for the prefix shared with the previous turn, so only new tokens are evaluated. It streams raw output back to Ruby every 50 ms, which elelem prints as thinking. When generation ends, llama.cpp’s chat parser splits the output into reasoning, reply, and tool calls. If the parser finds no tool call but tools were offered, the shim tries two fallbacks: a bare JSON object with a known tool name and arguments, then a gpt-oss style to=functions.NAME tag.

Limitations #

• The live stream is the model’s raw output, shown as thinking. The parsed reply appears when generation ends.
• Tool calls depend on the model’s chat template and llama.cpp’s parser. The fallbacks cover common cases, not all.
• One model per process. Switching models means restarting elelem.
• Thread count is not configurable. llama.cpp picks it from the CPU.
• The GPU offload default compares only the model file size with free memory. It does not count the KV cache, so a large context can still run out of GPU memory. Set ELELEM_GGUF_N_GPU_LAYERS if that happens.
• To change the backend, reinstall the gem with ELELEM_LLAMA_BACKEND set.
• If the reply runs out of tokens before any text or tool call, you get [no reply: ran out of output tokens before finishing].

Development #

git clone https://git.mokhan.ca/elelem/llama.git
cd llama
bin/setup
bin/compile
bin/test

bin/setup initializes the vendor/llama.cpp submodule, runs bundle install, and downloads a GGUF model from Hugging Face into ~/.agents/models if it is not already there. bin/compile runs the same build as gem install. The spec covers model discovery.

Environment #