ELELEM-EVALS(7)

Name

elelem-evals — Test the agent, not the model

Synopsis

git clone https://git.mokhan.ca/elelem/evals.git
elelem-evals [options]

Description

Runs scripted coding tasks against any elelem provider in throwaway directories and checks the results with shell commands, file contents, and the final reply. Not published to RubyGems.

Full documentation: elelem-evals.

Environment

ELELEM_PROVIDER (default llama): Provider to run against when -p is not given.

Requirements

Ruby 4.0 or later. bash, git, sqlite3, and ruby on PATH, for the bundled cases. A provider gem (elelem-llama by default).

Status

Alpha: works, with known gaps. Check the known issues first. Source only; not published to RubyGems. History: commits.

Bugs

Known issues: plugin:evals. Patches are welcome by email; see git-send-email.io.