A new CLI/Library for evals

Demonstrating a new CLI/library for configuration-based LLM evaluations. Run evaluations on prompts with datasets or existing model output.

Overview

I’d like to show off a new configuration-based evaluation system for LLMs that I’ve been working on. The project is a an evaluation CLI for running evaluations on prompts with datasets or for existing model output, either from a production system or from some other run mechanism.