Benchmarks

Use this endpoint to retrieve all benchmarks available to your account. Each benchmark defines a specific evaluation suite you can run against your model endpoint by submitting a job.

Endpoint

GET https://api.harmstack.com/v0/benchmarks

Authentication

string

required

Bearer YOUR_API_KEY

Request

No query parameters are required.

curl https://api.harmstack.com/v0/benchmarks \
  --header "Authorization: Bearer YOUR_API_KEY"

Response

Returns 200 OK with a JSON object containing a benchmarks array.

object[]

required

Array of benchmark objects available to your account.

Show benchmark fields

number

required

Unique numeric identifier for the benchmark. Use this value as benchmark_id when submitting jobs.

number

required

Internal identifier for the benchmark publisher.

string

required

Internal name of the benchmark (e.g. "haystack-v1").

string

required

Human-readable name of the benchmark (e.g. "Haystack Medical QA v1").

number

required

Version number of the benchmark.

number

ID of the associated benchmark property configuration, if set.

number

required

Credit cost per benchmark unit submitted.

string

required

Currency code for the price (e.g. "USD").

string

required

ISO 8601 timestamp of when the benchmark was created.

string

required

ISO 8601 timestamp of the last update.

{
  "benchmarks": [
    {
      "id": 1,
      "merchant_id": 3,
      "name": "haystack-medical-v1",
      "display_name": "Haystack Medical QA v1",
      "version": 1,
      "benchmark_property_id": 12,
      "price_per_benchmark_unit": 1.0,
      "currency": "USD",
      "created_at": "2024-01-15T10:00:00Z",
      "updated_at": "2024-06-01T08:30:00Z"
    }
  ]
}

Copy the id from a benchmark object and pass it as benchmark_id when submitting a job via POST /v0/jobs.

Overview

Endpoints

Endpoint

Authentication

Request

Response

​Endpoint

​Authentication

​Request

​Response

Endpoint

Authentication

Request

Response