ELELEM-LLAMA(7)

Name

elelem-llama — llama.cpp, in-process

Synopsis

gem install elelem-llama
elelem chat -p llama [--model name]

Description

Runs GGUF models inside the elelem process through llama.cpp. No server, no network call.

Full documentation: elelem-llama.

Environment

ELELEM_GGUF_MODEL: Path to a .gguf file. Unset means look in ~/.agents/models.
ELELEM_GGUF_N_CTX (default 0): Context size in tokens. 0 means the model’s trained context, capped at 65536.
ELELEM_GGUF_N_GPU_LAYERS (default auto): Layers to offload to the GPU. Auto picks 999 (all) or 0; see Configure.
ELELEM_GGUF_MAX_TOKENS (default 0): Output token limit per reply. 0 means the rest of the context window.
ELELEM_GGUF_TEMP (default 0.7): Sampling temperature. 0 or below means greedy sampling.
ELELEM_GGUF_SEED (default -1): Sampling seed.
ELELEM_TOP_K (default 40): Top-k sampling. Ignored when temperature is 0 or below.
ELELEM_TOP_P (default 0.95): Top-p sampling. Ignored when temperature is 0 or below.
ELELEM_REPEAT_LAST_N (default 0): Tokens considered by the repeat penalty. 0 disables the penalty.
ELELEM_REPEAT_PENALTY (default 1.0): Repeat penalty. 1.0 means no penalty.
ELELEM_KV_CACHE_TYPE (default q8_0): KV cache type. One of f16, f32, q4_0, q8_0; anything else means q8_0.
ELELEM_LLAMA_BACKEND (default auto): Build time only: auto, cpu, vulkan, cuda, hip, or metal.
ELELEM_HIP_ARCH (default detected, else gfx1100): Build time only. GPU target for the hip backend.
ELELEM_LOG_LEVEL (default warn): Also sets the llama.cpp log threshold (debug, info, warn, error).

Requirements

Ruby 4.0 or later. A C++17 compiler and CMake. Optional, for GPU builds: the CUDA toolkit, the Vulkan SDK, ROCm, or macOS with Metal. A GGUF model file.

Status

Ready: works today and is used daily. Published on RubyGems. History: commits.

Bugs

Known issues: plugin:llama. Patches are welcome by email; see git-send-email.io.