# Compare models with elelem-evals

> Run the same coding tasks against different providers and models, and write your own cases to test what matters to you.


At the end of this guide you will have run a suite of real coding tasks
against two models and compared the scores, and written a case of your
own. `elelem-evals` runs each task in a scratch copy of a small project,
lets the agent work, then checks the result with shell commands and text
matches. It reports pass or fail per case and nothing else: no timings, no
token counts.

> **Experimental:** `elelem-evals` 0.1.0 is not published on RubyGems.
> You run it from source.

## Before you start

- Ruby 4.0 or newer, Bundler, and git.
- A C/C++ toolchain and CMake. The repository's Gemfile includes
  `elelem-llama`, which compiles llama.cpp during `bundle install`.
- At least one working provider: an Ollama server with two models pulled,
  or GGUF files for `elelem-llama`. See
  [Run elelem fully offline](/guides/local-models/).

## 1. Get the source

```bash
git clone https://git.mokhan.ca/elelem/evals.git
cd evals
bin/setup
```

`bin/setup` runs `bundle install`. The bundle includes `elelem`,
`elelem-builtins`, `elelem-llama`, and `elelem-ollama`.

## 2. Fix the provider bug in 0.1.0

In 0.1.0, every case fails with `undefined method 'fetch' for an instance
of Proc` in the log, before the model is called. The runner never builds
the provider. Open `lib/elelem/evals/runner.rb` and change one line in
the private `provider` method:

```ruby
      def provider
        @built ||= @provider.call
      end
```

The original line is `@provider ||= @provider.call`, which returns the
unbuilt factory. Skip this step if your checkout already has the fix.

## 3. See what is in the suite

```bash
bundle exec exe/elelem-evals --list
```

```text
build/diagnose-failing-test
create/fizzbuzz-from-scratch
edit/read-before-write
edit/restructure-with-write
edit/multi-line-edit
...
```

There are 30 cases in 15 groups, one group per file in `cases/`: editing,
fixing a failing test, refactoring, git, SQL, shell, search, and others.
The `holdout` group holds paraphrased twins of other cases, kept apart so
a prompt tuned on the rest can be checked for overfitting.

Each case names a fixture, a small project in `fixtures/`. The runner
copies it to a temporary directory, runs the case's `setup` commands
there, sends the prompts, and checks the result. The directory is deleted
afterwards.

## 4. Run one group

```bash
bundle exec exe/elelem-evals -p ollama -c 'git/*'
```

`-p` picks the provider. Without it, `elelem-evals` uses `ELELEM_PROVIDER`
or, if that is unset, `llama`. `-c` filters cases with a shell glob on
`group/id`.

The output looks like this:

```text
PASS git/commit-50-72
FAIL git/last-commit-subject
  response missing "Add greeting script"

1/2 passed
```

Each failing case lists the checks that failed. The model's thinking text
is printed to stdout as well, so a `PASS` or `FAIL` may follow it on the
same line. The last line is always the summary. The exit status is 0 only
when every case passes.

## 5. Compare two models

`elelem-evals` takes no `--model` flag. Each provider reads its model from
its own environment variable, so set that per run:

```bash
for model in gpt-oss:latest qwen3:latest; do
  echo "== $model"
  OLLAMA_MODEL=$model bundle exec exe/elelem-evals -p ollama | tail -1
done
```

```text
== gpt-oss:latest
...
== qwen3:latest
...
```

For `elelem-llama`, use `ELELEM_GGUF_MODEL`:

```bash
ELELEM_GGUF_MODEL=~/.agents/models/a.gguf bundle exec exe/elelem-evals -p llama | tee a.log
ELELEM_GGUF_MODEL=~/.agents/models/b.gguf bundle exec exe/elelem-evals -p llama | tee b.log
grep -oE '(PASS|FAIL) [a-z]+/[a-z0-9-]+' a.log > a.txt
grep -oE '(PASS|FAIL) [a-z]+/[a-z0-9-]+' b.log > b.txt
diff a.txt b.txt
```

The `diff` shows the cases where the two models disagree, which tells you
more than the totals do. The provider is built once per run, so a GGUF
model loads once, not once per case.

Small suites are noisy. Run each model more than once before trusting a
difference of one or two cases.

## 6. Compare providers

To run against a hosted model, add its provider gem to the bundle:

```bash
bundle add elelem-anthropic
ANTHROPIC_API_KEY=... bundle exec exe/elelem-evals -p anthropic | tail -1
```

`elelem-openai` works the same way with `OPENAI_API_KEY`, and
`ANTHROPIC_MODEL` or `OPENAI_MODEL` pick the model.

## 7. Compare system prompts

`-P` replaces elelem's system prompt template for the run. It is an ERB
template, like elelem's default:

```bash
cat > terse.erb <<'EOF'
You are a terminal coding agent. Answer in as few words as the task allows.

<%= agents_md %>
EOF
bundle exec exe/elelem-evals -p ollama -P terse.erb | tail -1
```

`<%= agents_md %>` inserts `~/.agents/AGENTS.md` and the nearest
`AGENTS.md`, as in a normal session.

## 8. Write your own case

Keep your cases in a directory of your own:

```bash
mkdir -p my-evals/cases my-evals/fixtures/sums
printf '3\n4\n5\n' > my-evals/fixtures/sums/numbers.txt
```

Create `my-evals/cases/math.yml`:

```yaml
# Answer from a file without editing anything.
- id: sum-numbers
  fixture: sums
  turns:
    - what is the sum of the numbers in numbers.txt? reply with the number only.
  expect:
    response_contains: ["12"]
    response_matches: ['\A\D*12\D*\z']

# Write a file with the result, in at most 10 model calls.
- id: write-total
  fixture: sums
  turns:
    - write the sum of the numbers in numbers.txt to total.txt, nothing else
  expect:
    verify: test "$(cat total.txt)" = 12
    files:
      numbers.txt:
        contains: ["3", "4", "5"]
    max_turns: 10
```

The file name is the group (`math`). Each case has:

| Key | Meaning |
|---|---|
| `id` | Name within the group. |
| `fixture` | Directory under the fixtures directory to copy. |
| `setup` | Optional shell commands to run in the copy first, such as `git init -q`. A failing command fails the case. |
| `turns` | One or more prompts, sent in order in one session. |
| `expect` | The checks, below. |

And `expect` can hold:

| Key | Passes when |
|---|---|
| `verify` | The shell command exits 0 in the workspace. |
| `files` | Each listed file exists, has every `contains` string, and has no `not_contains` string. |
| `response_contains` | The final reply has every string. |
| `response_not_contains` | The final reply has none of the strings. |
| `response_matches` | The final reply matches every regular expression. |
| `max_turns` | Not a check: the most model calls allowed for the case. Default 30. |

A case that goes over `max_turns` ends with the reply
`Error: turn limit N exceeded` and is judged on what it left behind.

Run it:

```bash
bundle exec exe/elelem-evals -p ollama --cases-dir my-evals/cases --fixtures-dir my-evals/fixtures
```

`--cases-dir` replaces the built-in cases, and `--fixtures-dir` the
built-in fixtures, so pass both when your cases use your own fixtures.

## Things that affect results

- The agent gets the tools from `elelem-builtins` plus every plugin in
  `~/.agents/plugins/` and `.agents/plugins/` in the directory you run
  from. Your personal plugins change the scores. Move them aside for a
  clean comparison.
- Setup commands and the agent's own commands run outside Bundler's
  environment, so `ruby` and `git` in a fixture behave as they would in a
  fresh shell.
- Sampling is random. For `elelem-llama`, a fixed `ELELEM_GGUF_SEED` and
  a lower `ELELEM_GGUF_TEMP` reduce the variation between runs.

## What you built

A repeatable way to score providers, models, and system prompts on the
same coding tasks, and a directory of cases for the work you care about.

## Next steps

- [Write a provider plugin](/guides/write-a-provider/) and score it
  against the published ones.
- [Run elelem fully offline](/guides/local-models/) to try GGUF models.
- Read the [elelem-evals](/plugins/evals/) page.


Canonical: https://elelem.sh/guides/evals/