Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.harmstack.ai/llms.txt

Use this file to discover all available pages before exploring further.

Harmstack helps teams evaluate medical AI quality with repeatable, structured benchmarks. Point Harmstack at your model endpoint, run a benchmark, and review results through the harmstack CLI or API.

Quick Start

Install the CLI with Homebrew or a release archive, then run your first benchmark in minutes.

CLI Reference

Explore every command and flag in the harmstack CLI.

API Authentication

Learn how to authenticate API requests with your API key.

API Endpoints

Browse the full public REST API reference.

​
How it works

1

Get your API key

Obtain your HARMSTACK_API_KEY from Vetted Medical. This key authenticates all CLI commands and API requests.
2

Install the CLI

Prebuilt binaries are published on GitHub:vettedmedical/harmstack-install-v0 — latest releasemacOS (Apple Silicon) and Linux (amd64): install with Homebrew or download the .tar.gz for your platform. Windows (amd64): download the .zip and use harmstack.exe. Full steps are in Quick Start and CLI overview.
3

Configure your model endpoint

Set your model endpoint and credentials in the wizard, or export defaults so you do not need to re-enter them each time:
export TARGET_MODEL_ENDPOINT_URL="https://api.openai.com/v1/responses"
export TARGET_MODEL_API_KEY="sk-proj-1234567"
Harmstack supports multiple provider shapes, including openai, openai_responses, gemini, and raw.
4

Run and review benchmarks

Choose a benchmark, submit a run, and inspect job-level scores and aggregate stats from the CLI or API.

​
What you can do

  • Submit benchmarks - Run one-off or repeated evaluations against your model endpoint
  • Inspect results - Review scores, metadata, and run details for each job
  • Compare runs - Measure quality changes across model or prompt revisions
  • Track trends - Monitor benchmark performance over time
  • Automate in CI/CD - Run non-interactively with --consentandskip
Admin-only API endpoints are intentionally excluded from this documentation. If you need admin access, contact Vetted Medical support.
Start with Quick Start for installation options and a complete non-interactive command example.