Core concepts

Products, suites, cases, runs — and why QA status is not execution status.

AI product

The system under test: any AI-powered surface reachable over HTTP (chat assistant, RAG service, agent workflow, AI API).

Test suite

A versioned collection of test cases for one product. Runs and schedules attach to suites.

Test case

One scenario: input, expected behavior, category, severity, tags. Cases have versions; runs record which version executed.

Test run

One execution of a suite against the live product, producing result rows and a QA report.

Execution status vs QA status

A request can succeed (HTTP 200) and still fail quality checks. Execution status describes transport; QA status describes whether the answer passed semantic evaluation. HelixQA keeps them separate everywhere — reports, tables and metrics.

Failure types

Execution errors mean the product was unreachable or returned errors. Evaluation errors mean the judge/provider failed — they are never counted as product-quality failures and do not lower scores as failures.