Quick Start

This guide gets you from zero to a submitted benchmark run. For command and flag details, see the CLI overview.

Prerequisites

A Harmstack API key (HARMSTACK_API_KEY) from Vetted Medical
macOS (Apple Silicon), Linux (amd64), or Windows (amd64)

Install the CLI

Binaries and checksums live on GitHub: vettedmedical/harmstack-install-v0 — latest release

macOS (Apple Silicon) and Linux (amd64) with Homebrew

brew tap vettedmedical/harmstack-install-v0 https://github.com/vettedmedical/harmstack-install-v0
brew install harmstack
harmstack --help

Manual install from a release archive

Download the asset that matches your OS (version numbers differ per release):

macOS Apple Silicon: harmstack_<version>_darwin_arm64.tar.gz
Linux amd64: harmstack_<version>_linux_amd64.tar.gz
Windows amd64: harmstack_<version>_windows_amd64.zip (contains harmstack.exe)

macOS or Linux — extract and put harmstack on your PATH:

tar -xzf harmstack_*_darwin_arm64.tar.gz
# Linux amd64: use harmstack_*_linux_amd64.tar.gz
chmod +x ./harmstack
mkdir -p "$HOME/.local/bin"
mv ./harmstack "$HOME/.local/bin/harmstack"
export PATH="$HOME/.local/bin:$PATH"
harmstack --help

Windows — unzip the file, move harmstack.exe to a folder on your PATH, open a new terminal, then run harmstack --help. Optional: verify downloads using checksums.txt from the same release. More detail, upgrade paths, and platform notes: CLI overview — Installation.

Set your Harmstack API key

export HARMSTACK_API_KEY="your_api_key_here"

For a persistent shell setup (example: zsh):

echo 'export HARMSTACK_API_KEY="your_api_key_here"' >> ~/.zshrc
source ~/.zshrc

Configure your model endpoint (non-interactive)

Export the URL and API key for the model you are benchmarking:

export TARGET_MODEL_ENDPOINT_URL="https://api.openai.com/v1/responses"
export TARGET_MODEL_API_KEY="sk-proj-1234567"

Harmstack supports multiple provider shapes, including openai, openai_responses, gemini, and raw.

Run your first benchmark

This example skips the interactive wizard and submits a Haystack run from the root command:

harmstack --haystack --consentandskip \
  --provider openai_responses \
  --benchmark-id 2 \
  --unit-count 1

Adjust --benchmark-id, --unit-count, and --provider to match your setup. See Flags for all options.

Get Started

CLI Reference

API Reference

Prerequisites

Install the CLI

macOS (Apple Silicon) and Linux (amd64) with Homebrew

Manual install from a release archive

Set your Harmstack API key

Configure your model endpoint (non-interactive)

Run your first benchmark

Next steps

CLI overview

CLI commands

​Prerequisites

​Install the CLI

​macOS (Apple Silicon) and Linux (amd64) with Homebrew

​Manual install from a release archive

​Set your Harmstack API key

​Configure your model endpoint (non-interactive)

​Run your first benchmark

​Next steps

CLI overview

CLI commands

Prerequisites

Install the CLI

macOS (Apple Silicon) and Linux (amd64) with Homebrew

Manual install from a release archive

Set your Harmstack API key

Configure your model endpoint (non-interactive)

Run your first benchmark

Next steps