# elelem-evals

> Runs scripted coding tasks against any elelem provider in throwaway directories and checks the results with shell commands, file contents, and the final reply. Not published to RubyGems.

Maturity: alpha.  Source: https://src.mokhan.ca/elelem/evals/


`elelem-evals` answers one question: given this provider and
system prompt, does the agent finish real tasks? Each case copies a small
project into a temporary directory, sends the agent one or more prompts
with the `elelem-builtins` tools, then checks what changed. The
repository ships 30 cases in 15 groups. It is not on RubyGems; run it
from a clone.

## Install

```bash
git clone https://git.mokhan.ca/elelem/evals.git
cd evals
bin/setup
```

`bin/setup` runs `bundle install`. The Gemfile includes `elelem-llama`
and `elelem-ollama`. To evaluate another provider, add its gem to the
Gemfile first.

## Use

```bash
bundle exec exe/elelem-evals -p ollama
```

```text
PASS build/diagnose-failing-test
FAIL fix/fix-failing-test
  verify failed (exit 1): ruby test.rb
  lib/calc.rb missing "a + b"
ERROR git/commit-50-72: Elelem::Evals::BoundedProvider::TurnLimitExceeded: turn limit 30 exceeded

1/3 passed
```

Each case prints `PASS`, `FAIL` with one line per failed check, or
`ERROR` if it raised. The last line is the pass count. The exit status is
`0` only when every case passed, so it works in CI.

```text
Usage: elelem-evals [options]
    -p, --provider NAME              Provider to run against (default: llama)
    -c, --cases GLOB                 Filter cases by group/id glob, e.g. build/*
        --cases-dir DIR              Directory of case yaml files
        --fixtures-dir DIR           Directory of fixture directories
    -P, --prompt FILE                System prompt template file
    -l, --list                       List matching cases without running them
```

Run one group, list cases, or try a different system prompt:

```bash
bundle exec exe/elelem-evals -p ollama -c 'search/*'
bundle exec exe/elelem-evals -l
bundle exec exe/elelem-evals -p ollama -P my-prompt.erb
```

`-P` replaces elelem's system prompt template with the file you give,
which is how you compare 2 prompts on the same cases. The `-c` glob is
matched against `group/id`.

## Write a case

Cases live in `cases/*.yml`. The file name is the group. Each file is a
list of cases:

```yaml
- id: fix-failing-test
  fixture: broken
  setup:
    - git init -q
  turns:
    - run test.rb and fix whatever is broken
  expect:
    verify: ruby test.rb
    files:
      lib/calc.rb:
        contains: ["a + b"]
        not_contains: ["a - b"]
    response_contains: ["add"]
    response_not_contains: ["sorry"]
    response_matches: ['\A[\s\S]{0,400}\z']
    max_turns: 30
```

| Key | Meaning |
|---|---|
| `id` | name within the group |
| `fixture` | directory under `fixtures/` to copy into the workspace |
| `setup` | shell commands run in the workspace before the agent starts; any failure is an error |
| `turns` | prompts sent in order; the reply to the last one is checked |
| `expect.verify` | shell command that must exit `0` |
| `expect.files` | per file, strings it must `contains` and must not `not_contains` |
| `expect.response_contains` | strings the final reply must include |
| `expect.response_not_contains` | strings it must not include |
| `expect.response_matches` | Ruby regular expressions it must match |
| `expect.max_turns` | maximum model requests for the whole case, default 30 |

Only `id`, `fixture`, and `turns` are required. A fixture is a plain
directory; add one under `fixtures/` for a new case. Use `--cases-dir`
and `--fixtures-dir` to keep your own suite outside the repository.

## How it works

For each case, the runner creates a directory with `Dir.mktmpdir`, copies
the fixture into it, and runs the `setup` commands with `bash -c`. It then
changes into that directory, builds a fresh session with the provider,
and applies every setup block, which adds the `elelem-builtins` tools and
anything from your plugin files. Each turn goes through the normal agent
loop.

The provider is wrapped so that each request to the model counts as one
turn. Going over `max_turns` raises, and the case is reported as an
`ERROR`. Checks run after the last turn. The workspace is deleted when
the case ends, pass or fail.

## Limitations

- Cases run one at a time, with no time limit beyond `max_turns`.
- The agent runs with the full builtin tools, including shell commands,
  and is not sandboxed. The temporary directory is the working directory,
  not a boundary.
- Plugin files in `~/.agents/plugins/` are loaded too, so your local
  plugins can change results.
- Every check is a string, regex, or exit status. There is no scoring by
  a model and no partial credit.
- Results are printed only. Nothing is saved for comparing runs.
- The repository has no specs yet.

## Development

```bash
git clone https://git.mokhan.ca/elelem/evals.git
cd evals
bin/setup
bundle exec exe/elelem-evals -l
```

The gem depends on `elelem` 0.13 or later and `elelem-builtins`.


## Environment

| Variable | Default | Description |
|---|---|---|
| `ELELEM_PROVIDER` | `llama` | Provider to run against when -p is not given. |

Canonical: https://elelem.sh/plugins/evals/