Promptfoo for LLM Evaluation: Regression Checks Explained

Promptfoo provides a CLI and library for comparing and evaluating LLM applications, with automated scoring and CI/CD integration. The official introduction emphasizes benchmarks specific to a use case. That specificity is the important part: a repeatable evaluation only helps when its examples and criteria reflect failures that matter to the application.

Promptfoo: the documented flow

Loading diagram…

Conceptual map of the documented components, not a measured deployment or proof of failure recovery.

Choose a decision before choosing a score

A useful evaluation asks whether a change should ship, whether one model fits a workload better, or whether a known failure has been addressed. Promptfoo documents side-by-side matrix views across prompts and inputs and support for automatically scoring outputs. Source

For example, a support assistant might need to preserve a policy boundary while answering a question accurately. Those are two separate requirements. One aggregate score can hide a regression in one while rewarding improvement in the other.

The useful design move is to define the decision and its acceptance conditions first. Then choose examples that could change that decision, including cases where an apparently fluent answer is still wrong.

Control what changes in a comparison

Promptfoo supports model and prompt comparisons. Source A comparison becomes easier to interpret when the input cases and criteria stay stable while the intended variable changes.

If the prompt, provider, test data, and scoring rule all change together, a different score does not explain which change helped. This is a general evaluation limitation, not something a matrix display can remove. Keep the configuration and result together so later readers can understand what was compared.

Match the check to the property you need

The official introduction documents automatic metrics and evaluation of prompts, models, and applications. Source Some properties have a clear mechanical definition; others require judgment about meaning.

Property to assess	Useful evaluation approach	Limitation to remember
A required output shape	A precise deterministic check	Valid shape does not establish factual accuracy
Whether an answer addresses a question	A clearly defined semantic criterion	A model judge can make mistakes
Behavior on a known failure	A representative regression example	Passing it does not establish universal coverage

Treat these as complementary questions. Adding more checks helps only if they test distinct requirements rather than repeat the same weak proxy.

A release gate is different from a red-team exercise

Promptfoo documents both CI/CD checks and automated red teaming. Source A regression gate asks whether an expected behavior changed. A red-team exercise searches for ways a system can fail under adversarial conditions.

Both can inform a release decision, but a clean report is evidence about the cases exercised, not a guarantee that no vulnerabilities remain. Keep severe failures visible rather than averaging them away into an otherwise high score.

Local execution is not the same as local model processing

The README describes local evaluation execution and separately instructs users to configure model-provider API keys. Source Running the CLI locally therefore does not establish that prompts never leave the machine: that depends on the configured providers and integrations.

Use the documented setup for the chosen runtime, then build a small, representative evaluation around the application decision. Expand it when failures reveal missing coverage. Promptfoo supplies repeatable execution and comparison; the usefulness of the result comes from the questions encoded in the evaluation.

A release gate for a refund-policy prompt change

Consider a hypothetical prompt change intended to make a support assistant more helpful. A release gate should check whether it improved answers while preserving policy boundaries, rather than rewarding longer or more confident responses. Promptfoo's repeatable comparisons and automated checks provide the execution mechanism. Evaluation workflow

Test case	Required outcome	Separate failure signal
An eligible refund request	Explain the documented process accurately	Wrong eligibility or invented steps
A request outside policy	Explain the actual limit without inventing an exception	An unauthorized promise
Missing order facts	Ask for the information needed to decide	A fluent answer built on guessed facts

Use fixed case inputs and criteria for the baseline and candidate prompts. Record the provider, model, prompt and judge configuration so the comparison can be interpreted later. A hard policy failure should block the candidate even if its average helpfulness score improves.

Add adversarial cases as a separate exercise when the threat model calls for them; regression coverage and red teaming answer different questions. The table is an example acceptance design, not an executed Promptfoo suite or a universal rubric for support systems.

Evidence and scope

Primary repository documentation was refreshed on 14 October 2026. The worked scenario is a planning exercise derived from documented behavior, not a product test. Reported measurements remain attributed to their source and workload.