Core Concepts

The key building blocks of Avaliar — read this before diving into the detailed guides

Avaliar is built around a small set of core primitives. This page gives you a quick mental model of each concept. Each section links to the full documentation where you can go deeper.

Traces

A trace represents a single LLM interaction from input to output. Traces are the fundamental unit of observability in Avaliar. Every function decorated with @traceable (or every request routed through the Proxy) creates a trace that captures the full request-response lifecycle — including the prompt, completion, model, provider, latency, token counts, and any detected issues.

Traces are stored in your organization’s workspace and visible in the Traces dashboard. You can filter, search, and inspect them to understand how your LLM application behaves in production.

→ Traces on the Platform

Spans

Spans are individual operations within a trace. A trace can contain multiple nested spans in a parent-child hierarchy, letting you see exactly how a complex LLM pipeline executes step by step.

Avaliar supports four span types:

Type	Description
llm	A direct call to a language model
tool	A tool or function call made by the model
agent	An autonomous agent step that may contain child spans
generic	Any other operation you want to track

Nesting spans lets you trace multi-step workflows such as RAG pipelines, agent loops, or chain-of-thought sequences. Parent-child relationships are tracked automatically — no manual wiring needed.

→ Traceable Decorator

Detection

Detection is the automated analysis of traces for safety issues. Avaliar provides six built-in detectors:

Detector	What It Finds
Prompt Injection	Attempts to override system instructions via user input
Jailbreak	Techniques designed to bypass model safety guardrails
Toxicity	Harmful, abusive, or offensive language in inputs or outputs
PII	Personally identifiable information — emails, phone numbers, SSNs, etc.
Bias	Stereotyping, demographic bias, or unfair treatment in model outputs
Hallucination	Factually incorrect or fabricated information in model responses

Detection runs in one of two modes — local (data stays on your infrastructure) or cloud (higher throughput, zero ops overhead).

→ Detection Overview · Detector Reference · Detection Modes

Issues

An issue is a finding produced by a detector during trace analysis. Each issue contains:

• Type — the detector that found it (e.g., toxicity, pii)
• Severity — one of low, medium, high, or critical
• Confidence — a score from 0 to 1 indicating certainty
• Description — a human-readable explanation of the finding
• Excerpt — the specific portion of the input or output that triggered detection

Issues are attached to their parent trace and visible in both the trace detail view and the aggregated Issues section of the dashboard.

Benchmarks & Evals

Benchmarks are standardized tests that measure an LLM’s general capabilities across well-known academic datasets:

Benchmark	Measures
MMLU	Broad multi-task knowledge across 57 subjects
DROP	Discrete reasoning over paragraphs
HellaSwag	Commonsense NLI and sentence completion
TruthfulQA	Tendency to generate truthful vs. imitative-falsehood answers
BigBenchHard	Multi-step reasoning tasks
HumanEval	Functional code generation correctness

Evals are safety-focused evaluations that measure bias and harm rather than general capability:

Eval	Focus
BBQ	Social bias across 11 demographic categories
BOLD	Bias and toxicity in open-ended text generation
HExPHI	Whether models follow harmful instructions
RealToxicityPrompts	Likelihood of generating toxic continuations

→ Benchmarks Overview · Available Benchmarks · Evals Overview

Alerts

Alerts are automated notifications triggered when issues meet conditions you define. Each alert rule specifies:

• Condition type — threshold (count exceeds N), trend (rate increasing), pattern (repeated issue type), or anomaly (statistical outlier)
• Channels — where to send the notification: email, Slack, or webhook

You can scope alerts to specific detectors, severity levels, models, or environments. Full alert history is stored and available for auditing.

→ Alerts on the Platform

Reports

Reports are generated documents that aggregate trace data, issues, and benchmark results into a structured format for compliance and stakeholder review.

Report	Purpose
Security Risk	Detected threats, attack patterns, and mitigation status
Model Cost	Token usage, latency, and spend by model and provider
Platform Ops	System health, uptime, trace volume, and error rates
AI Risk Posture	Overall safety score combining detection, benchmark, and eval results

Reports can be generated on-demand or scheduled, and exported as PDF or JSON.

→ Reports on the Platform