Skip to main content

Documentation Index

Fetch the complete documentation index at: https://chatjs.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

ChatJS uses Evalite to evaluate AI agent outputs. Evaluations help you measure response quality, catch regressions, and iterate on prompts with confidence.

​
How It Works

Evaluations follow a simple structure:
  1. Data: Test cases with inputs and expected outputs
  2. Task: The AI operation being evaluated
  3. Scorers: Functions that grade the output
Results are stored in SQLite and displayed in a web UI.

​
Running Evaluations

# Watch mode (re-runs on file changes)
bun eval:dev

# View results UI
bun eval:serve

​
Writing Evaluations

Create files with the .eval.ts extension in the evals/ directory. See existing evals for examples. Use runCoreChatAgentEval from lib/ai/eval-agent.ts to wrap the core chat agent for evaluation contexts.

​
Custom Scorers

Scorers receive the output, expected value, and input. Return a number between 0 and 1. Evalite also provides built-in scorers like exactMatch, answerCorrectness, and answerRelevancy.