Promptfoo.

TL;DR — Promptfoo is a test harness for LLM apps that runs prompts, assertions, and regression suites across models before shipping changes.

What it is

Promptfoo is a CLI-first evaluation framework. You define test cases and assertions in YAML, then execute batch runs against one or many models.

Why it exists

Teams need repeatable pass/fail checks for model behavior, cost, safety, and latency. Promptfoo turns ad-hoc prompt checks into CI-friendly tests.

Install

npm install -g promptfoo
promptfoo init

Basic usage

from promptfoo import evaluate
# define test suites in promptfooconfig.yaml
# run in CI: promptfoo eval

When to use, when to skip

Use it when this category is a bottleneck in your agent stack and you want faster delivery with fewer custom components.

Skip it when your workload is tiny, requirements are fixed, or a plain provider SDK plus a few local functions is enough.

Alternatives

Compare with adjacent tools in the same AI Native category and choose based on interface style, deployment model (hosted vs self-hosted), and team familiarity.

Verified against project documentation, June 2026.