Everything you need to Test AI Properly

From product description to executable suites

Everything starts with an AI product: the assistant, RAG service, copilot or agent endpoint you want under test. Register it once, then build suites that version with your prompts. Write cases by hand when you know the failure modes — or generate candidates from the product description and supporting docs.

Generation never writes to your suite uninvited. Candidates arrive as a preview list with suggested categories and severities; selecting which ones to keep is an explicit approval. The result is a suite that reflects how your product actually fails, not a pile of auto-generated noise.

Deterministic gates plus semantic judges

A run executes every enabled case against the live product. Transport problems are recorded as execution errors; judge or provider failures are recorded as evaluation errors — and neither is allowed to masquerade as a product-quality failure. Scores come only from real evaluations.

Deterministic checks answer the cheap questions fast: valid JSON, required fields, latency under budget, banned content absent. LLM-based judges handle the expensive ones — correctness, relevance, groundedness against provided context, instruction following, safety — each with a written reason you can audit instead of a bare number.

QA reports built for release decisions

Every completed run produces a report: overall score, pass rate, per-dimension breakdown, critical and high-severity failure counts, and per-test failure reasons with the judge's rationale. Execution and QA status sit in separate columns everywhere they appear — because a 200 response is not a passing test.

When you are about to ship, compare the candidate run against any baseline. The diff surfaces tests that flipped from pass to fail, score movement by dimension, and the specific reasons — labeled by metric, never by color alone. Regression detection is part of the product surface, not a spreadsheet exercise.

Schedules, CI and webhooks

AI products drift without anyone touching them: knowledge sources go stale, models get quieter upgrades, third-party tools change shape. Attach a cron schedule to a suite and HelixQA keeps running the same scenarios nightly or weekly, with next-run and last-run visible so you can trust the cadence.

For release-time gating, trigger runs from CI and subscribe webhooks for run.completed, run.failed and regression.detected. Your pipeline decides what 'good enough to ship' means — pass rate, overall score, zero critical failures — and blocks on evidence instead of intuition.

See it on your own AI product