> For a complete page index of the Captain API documentation, fetch https://docs.captain.dev/llms.txt?excludeSpec=true

# Queue Evaluation

POST https://api.captain.dev/v3/collections/{collection_name}/evals
Content-Type: application/json

Queue an evaluation of an uploaded case set against one to eight named query configurations. `Idempotency-Key` is required: within 24 hours the same key with the same body returns the stored `201` body with status `200`; the same key with a different body is `409` (`IDEMPOTENCY_KEY_REUSE`). Gold is resolved at accept time (document id, then unique basename); unresolved cases are `error` and never queried or billed. Each `configs[]` entry is the [v3 query](/reference/v3/query) body minus `query` and is validated the same way. Each unit (one question under one config) is billed as one query, so whether it costs credits depends on the plan; only units that ran (scored, or timed out at the 30 s limit) are counted. Scoring, limits, billing and the NDJSON format are on the [Evaluations](/guides/evaluations) guide.

Reference: https://docs.captain.dev/reference/evals/create

## Request

### Path parameters

- `collection_name` (string, required)

### Headers

- `Idempotency-Key` (string, required, nullable) — Caller-chosen key, 1 to 256 characters of `A-Z a-z 0-9 . _ : / = + -`, scoped to the organization for 24 hours.

### Body (application/json)

This endpoint expects a CreateEvalRequestV3.

- `configs` (list of EvalConfigV3, required)
- `upload_id` (string, required)
- `environment` (string, optional, nullable) — Echo-only label for the scorecard. Does not switch collections.

## Response

### 200

Idempotent replay: the stored original `201` body (including the original `eval_id`), not live status. Poll [Get Eval](/reference/evals/get) for progress.

- `collection_name` (string, required)
- `configs` (list of ResolvedConfigV3, required)
- `created_at` (string, required)
- `eval_id` (string, required)
- `idempotency_key` (string, required)
- `items_page` (EvalItemsPageV3, required)
- `preview` (EvalPreviewV3, required)
- `progress` (EvalProgressV3, required)
- `request_id` (string, required)
- `status` (enum, required)
  - Allowed values: `pending`, `running`, `completed`, `completed_with_errors`, `failed`
- `updated_at` (string, required)
- `upload_id` (string, required)
- `billing` (EvalBillingV3, optional, nullable)
- `completed_at` (string, optional, nullable)
- `environment` (string, optional, nullable)
- `error_code` (string, optional, nullable)
- `error_message` (string, optional, nullable)
- `items` (list of EvalItemV3, optional)
- `scorecards` (map from string to EvalScorecardV3, optional)
- `started_at` (string, optional, nullable)

### 201

Eval accepted and queued. `preview.billable_units` is the upper bound of the charge.

- `collection_name` (string, required)
- `configs` (list of ResolvedConfigV3, required)
- `created_at` (string, required)
- `eval_id` (string, required)
- `idempotency_key` (string, required)
- `items_page` (EvalItemsPageV3, required)
- `preview` (EvalPreviewV3, required)
- `progress` (EvalProgressV3, required)
- `request_id` (string, required)
- `status` (enum, required)
  - Allowed values: `pending`, `running`, `completed`, `completed_with_errors`, `failed`
- `updated_at` (string, required)
- `upload_id` (string, required)
- `billing` (EvalBillingV3, optional, nullable)
- `completed_at` (string, optional, nullable)
- `environment` (string, optional, nullable)
- `error_code` (string, optional, nullable)
- `error_message` (string, optional, nullable)
- `items` (list of EvalItemV3, optional)
- `scorecards` (map from string to EvalScorecardV3, optional)
- `started_at` (string, optional, nullable)

## Errors

### 400 Bad Request Error

`INVALID_ID_PREFIX`, missing or invalid `Idempotency-Key`, upload not yet PUT or expired, duplicate config names, a config failing v3 query validation, or a whole-file NDJSON problem (blank line, invalid JSON, duplicate case `id`, more than 10,000 lines, more than 80,000 units).

- `error` (string, required) — Error code, for example `INVALID_ID_PREFIX`. `HTTP_ERROR` when the route raised a plain message.
- `message` (string, required) — What went wrong.
- `path` (string, required) — The request path.
- `status_code` (integer, optional) — The HTTP status code. Present only on `HTTP_ERROR`.

### 401 Unauthorized Error

Missing or invalid authentication.

- `error` (string, required) — Error kind. `HTTP_ERROR` for errors raised by the route.
- `message` (string, required) — What went wrong.
- `path` (string, required) — The request path.
- `status_code` (integer, optional) — The HTTP status code.

### 403 Forbidden Error

API key does not have the query permission for this collection.

- `error` (string, required) — Error kind. `HTTP_ERROR` for errors raised by the route.
- `message` (string, required) — What went wrong.
- `path` (string, required) — The request path.
- `status_code` (integer, optional) — The HTTP status code.

### 404 Not Found Error

Collection not found (or an eval / upload that belongs to another organization: no existence oracle).

- `error` (string, required) — Error kind. `HTTP_ERROR` for errors raised by the route.
- `message` (string, required) — What went wrong.
- `path` (string, required) — The request path.
- `status_code` (integer, optional) — The HTTP status code.

### 409 Conflict Error

`IDEMPOTENCY_KEY_REUSE` (same key, different body) or `UPLOAD_ALREADY_BOUND` (upload already used by another eval).

- `error` (string, required) — Error code, for example `INVALID_ID_PREFIX`. `HTTP_ERROR` when the route raised a plain message.
- `message` (string, required) — What went wrong.
- `path` (string, required) — The request path.
- `status_code` (integer, optional) — The HTTP status code. Present only on `HTTP_ERROR`.

### 429 Too Many Requests Error

`TOO_MANY_PENDING_EVALS`: the organization already has 20 pending evals (`Retry-After` is set). Evals are paced as jobs, so this is never an org-wide query 429.

- `error` (string, required) — Error code, for example `INVALID_ID_PREFIX`. `HTTP_ERROR` when the route raised a plain message.
- `message` (string, required) — What went wrong.
- `path` (string, required) — The request path.
- `status_code` (integer, optional) — The HTTP status code. Present only on `HTTP_ERROR`.

## Types

### EvalConfigV3

`configs[]` = the Query v3 request body minus `query`, plus a name.

- `name` (string, required)
- `boost` (list of BoostClauseV3, optional, nullable)
- `exclude_chunk_types` (list of string, optional, nullable)
- `filter` (EvalConfigV3Filter, optional, nullable)
- `include` (QueryIncludeV3, optional) — Optional expansions for v3 query results. Leave expensive context off unless the caller needs it.
- `limit` (integer, optional, default: 10)
- `max_chunks_per_document` (integer, optional, nullable)
- `relation_direction` (enum, optional, default: outgoing)
  - Allowed values: `outgoing`, `incoming`, `both`
- `relation_types` (list of string, optional, nullable)
- `rerank` (EvalConfigV3Rerank, optional, default: false)
- `semantic_ratio` (double, optional, default: 0.5)

### ResolvedConfigV3

- `explicit_fields` (list of string, required)
- `name` (string, required)
- `query` (ResolvedQueryConfigV3, required) — Every Query v3 field, defaults filled in, rerank always in object form.

### EvalItemsPageV3

- `limit` (integer, required)
- `total` (integer, required)
- `next_cursor` (string, optional, nullable)

### EvalPreviewV3

- `billable_units` (integer, required)
- `cases` (integer, required)
- `cases_error` (integer, required)
- `configs` (integer, required)
- `units` (integer, required)

### EvalProgressV3

- `cases_error` (integer, required)
- `cases_total` (integer, required)
- `configs_failed` (integer, required)
- `configs_total` (integer, required)
- `elapsed_seconds` (integer, required)
- `percent` (integer, required)
- `units_completed` (integer, required)
- `units_error` (integer, required)
- `units_failed` (integer, required)
- `units_total` (integer, required)
- `eta_seconds` (integer, optional, nullable)
- `qps` (double, optional, nullable)

### EvalBillingV3

- `billable_units` (integer, required)
- `skipped_units` (integer, required)
- `failed_configs` (list of string, optional)

### EvalItemV3

- `expected_files` (list of string, required)
- `id` (string, required)
- `query` (string, required)
- `status` (enum, required)
  - Allowed values: `pending`, `running`, `scored`, `error`
- `error_code` (string, optional, nullable)
- `error_message` (string, optional, nullable)
- `expected_document_ids` (list of string, optional, nullable)
- `filters` (EvalItemV3Filters, optional, nullable)
- `results` (map from string to EvalUnitResultV3, optional)

### EvalScorecardV3

- `chunk_count` (integer, required)
- `collection_name` (string, required)
- `document_count` (integer, required)
- `error` (integer, required)
- `explicit_fields` (list of string, required)
- `failed` (integer, required)
- `name` (string, required)
- `query` (ResolvedQueryConfigV3, required) — Every Query v3 field, defaults filled in, rerank always in object form.
- `question_set_size` (integer, required)
- `scored` (integer, required)
- `status` (enum, required)
  - Allowed values: `scored`, `error`
- `environment` (string, optional, nullable)
- `error_code` (string, optional, nullable)
- `error_message` (string, optional, nullable)
- `latency_p50_ms` (integer, optional, nullable)
- `latency_p95_ms` (integer, optional, nullable)
- `mrr` (double, optional, nullable)
- `ndcg_at_10` (double, optional, nullable)
- `recall_at_1` (double, optional, nullable)
- `recall_at_10` (double, optional, nullable)
- `recall_at_3` (double, optional, nullable)

### BoostClauseV3

One boost rule. A boost rule is one entry in the request's `boost` array: some metadata to look for, and how much to favour the chunks carrying it. Each rule is exactly one of three shapes, and anything else returns a 422: * `{ "field": ..., "eq": ..., "weight": ... }` - one metadata value * `{ "field": ..., "in": [...], "weight": ... }` - any of several values, and list-valued metadata * `{ "chunk_ids": [...], "weight": ... }` - specific chunks, by id `weight` is always required. The other properties are marked optional only because which ones apply depends on the shape: `field` is required with `eq` or `in`, and unused by a `chunk_ids` rule.

- `weight` (double, required) — Required. Multiplier on the matching chunk's retrieval score, 0.2 to 5.0. The rule's retrieval pass is what makes a chunk appear; the weight decides where it lands. 1.5 to 2.0 is a good default and is usually enough to lift a tagged chunk to the top. Use 3.0 to 5.0 when boosted chunks compete with each other, or when a rule matches many chunks. Below 1.0 demotes, and only reorders chunks the query already found. When several rules match the same chunk their weights multiply, capped at 5.0.
- `field` (string, optional, nullable) — Required with `eq` or `in`; omit it for a `chunk_ids` rule. One of your own custom_metadata keys, the same keys `filter` accepts. Anything filterable is boostable, with no re-indexing. Captain's internal attributes (file_id, job_id, chunk_index and similar) return a 400.
- `eq` (BoostClauseV3Eq, optional, nullable) — Requires `field`. Boost chunks whose `field` equals this value. Strings, numbers, and booleans are supported. A list-valued field matches when any element equals it. Use `in` instead for several values.
- `in` (list of BoostClauseV3InItems, optional, nullable) — Requires `field`. Boost chunks whose `field` matches any of these values, up to 50. Use it for one key with several acceptable values, and for list-valued metadata.
- `chunk_ids` (list of string, optional, nullable) — Use instead of `field`/`eq`/`in` to boost specific chunks, up to 100. Accepts a `chunk_id` from a previous response, or `document_id:chunk_index`, which keeps resolving after the document is re-indexed. Use it to keep a follow-up question anchored to passages an earlier answer cited.
- `reserve` (integer, optional, default: 0) — With reranking on, guarantee this many of this rule's chunks a place on the page, chosen by reranker order. Reranking runs after boosting and can otherwise drop a boosted chunk whose text does not resemble the query. 0 (default) leaves the reranker's judgement in charge. The sum across all boost rules cannot exceed `limit`.

### EvalConfigV3Filter

### QueryIncludeV3

Optional expansions for v3 query results. Leave expensive context off unless the caller needs it.

- `document` (boolean, optional, default: true) — Include the parent document object for each returned chunk.
- `metadata` (boolean, optional, default: true) — Include Captain-generated retrieval metadata and application-supplied chunk metadata.
- `regions` (boolean, optional, default: false) — Include extracted layout regions, including bounding boxes when available.
- `relations` (boolean, optional, default: false) — Include graph relations connected to each returned chunk.
- `related_chunks` (boolean, optional, default: false) — Include linked chunks for returned relations.
- `neighboring_chunks` (boolean, optional, default: false) — Include the chunk immediately before and after each result in the source file.
- `document_metadata` (boolean, optional, default: false)
- `archived` (boolean, optional, default: false) — Include chunks archived by a sync `archive` deletion policy. Archived content is excluded from search by default; set true to surface it.

### EvalConfigV3Rerank

### ResolvedQueryConfigV3

Every Query v3 field, defaults filled in, rerank always in object form.

- `limit` (integer, required)
- `rerank` (ResolvedRerankV3, required)
- `semantic_ratio` (double, required)
- `boost` (list of ResolvedQueryConfigV3BoostItems, optional)
- `exclude_chunk_types` (list of string, optional)
- `filter` (ResolvedQueryConfigV3Filter, optional, nullable)
- `include` (map from string to boolean, optional)
- `max_chunks_per_document` (integer, optional, nullable)
- `relation_direction` (string, optional, default: outgoing)
- `relation_types` (list of string, optional, nullable)

### EvalItemV3Filters

### EvalUnitResultV3

- `status` (enum, required)
  - Allowed values: `pending`, `running`, `scored`, `failed`
- `error_code` (string, optional, nullable)
- `error_message` (string, optional, nullable)
- `hit` (boolean, optional, nullable)
- `latency_ms` (integer, optional, nullable)
- `rank` (integer, optional, nullable)
- `retrieved_document_ids` (list of string, optional, nullable)

### BoostClauseV3Eq

Requires `field`. Boost chunks whose `field` equals this value. Strings, numbers, and booleans are supported. A list-valued field matches when any element equals it. Use `in` instead for several values.

### BoostClauseV3InItems

### RerankOptions

Object form of the `rerank` parameter. The boolean form stays valid: `true` is equivalent to sending this object with every field at its default.

- `enabled` (boolean, optional, default: true) — Whether to rerank. Sending the object without this field means enabled — the object form exists to tune reranking.
- `candidate_limit` (integer, optional, nullable) — How many fused retrieval candidates are fetched and reranked before the top `limit` results are returned. Defaults to `limit` x 3 (the measured configuration). Must be >= `limit`; capped at 200. Larger pools can lift recall on corpora with many near-duplicate documents, at the cost of rerank latency.
- `model` (string, optional, nullable) — Reranker model. Voyage cross-encoders: `voyage-rerank-2.5` (default) and `voyage-rerank-3` (aliases `rerank-2.5`, `rerank-3`). Gemini LLM rerankers: `gemini-2.5-flash` and `gemini-3.8-flash` — higher quality ceiling on complex queries, higher latency; best with small candidate pools. The family aliases `voyage` and `gemini` resolve to each family's default. Unknown values return a 400 listing the allowed set.

### ResolvedRerankV3

- `enabled` (boolean, required)
- `candidate_limit` (integer, optional, nullable)
- `model` (string, optional, nullable)

### ResolvedQueryConfigV3BoostItems

### ResolvedQueryConfigV3Filter

## Examples

**Request**

```json
{
  "configs": [
    {
      "name": "baseline"
    },
    {
      "name": "rerank-deep",
      "exclude_chunk_types": [
        "page_header",
        "page_footer",
        "footnote"
      ],
      "filter": {
        "review_status": {
          "$in": [
            "approved",
            "in_review"
          ]
        }
      },
      "limit": 10,
      "max_chunks_per_document": 2,
      "rerank": {
        "enabled": true,
        "candidate_limit": 40,
        "model": "voyage-rerank-2.5"
      },
      "semantic_ratio": 0.35
    }
  ],
  "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
  "environment": "development"
}
```

**Response**

```json
{
  "collection_name": "customer_forms",
  "configs": [
    {
      "explicit_fields": [],
      "name": "baseline",
      "query": {
        "limit": 10,
        "rerank": {
          "enabled": false,
          "candidate_limit": null,
          "model": null
        },
        "semantic_ratio": 0.5,
        "boost": [],
        "exclude_chunk_types": [],
        "filter": null,
        "include": {
          "document": true,
          "metadata": true,
          "regions": false,
          "relations": false,
          "document_metadata": false,
          "related_chunks": false,
          "neighboring_chunks": false,
          "archived": false
        },
        "max_chunks_per_document": null,
        "relation_direction": "outgoing",
        "relation_types": null
      }
    },
    {
      "explicit_fields": [
        "exclude_chunk_types",
        "filter",
        "limit",
        "max_chunks_per_document",
        "rerank",
        "semantic_ratio"
      ],
      "name": "rerank-deep",
      "query": {
        "limit": 10,
        "rerank": {
          "enabled": true,
          "candidate_limit": 40,
          "model": "voyage-rerank-2.5"
        },
        "semantic_ratio": 0.35,
        "boost": [],
        "exclude_chunk_types": [
          "page_header",
          "page_footer",
          "footnote"
        ],
        "filter": {
          "review_status": {
            "$in": [
              "approved",
              "in_review"
            ]
          }
        },
        "include": {
          "document": true,
          "metadata": true,
          "regions": false,
          "relations": false,
          "document_metadata": false,
          "related_chunks": false,
          "neighboring_chunks": false,
          "archived": false
        },
        "max_chunks_per_document": 2,
        "relation_direction": "outgoing",
        "relation_types": null
      }
    }
  ],
  "created_at": "2026-09-12T17:30:01Z",
  "eval_id": "eval_019b4c2f11d04c8e9a7b3d5c2e1840ff",
  "idempotency_key": "eval-payments-2026-09-12-a",
  "items_page": {
    "limit": 100,
    "total": 3,
    "next_cursor": null
  },
  "preview": {
    "billable_units": 6,
    "cases": 3,
    "cases_error": 0,
    "configs": 2,
    "units": 6
  },
  "progress": {
    "cases_error": 0,
    "cases_total": 3,
    "configs_failed": 0,
    "configs_total": 2,
    "elapsed_seconds": 0,
    "percent": 0,
    "units_completed": 0,
    "units_error": 0,
    "units_failed": 0,
    "units_total": 6,
    "eta_seconds": null,
    "qps": null
  },
  "request_id": "req_20260912_eval001",
  "status": "pending",
  "updated_at": "2026-09-12T17:30:01Z",
  "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
  "billing": null,
  "completed_at": null,
  "environment": "development",
  "error_code": null,
  "error_message": null,
  "items": [],
  "scorecards": {},
  "started_at": null
}
```

**SDK Code**

```python Python REST
import requests

BASE_URL = "https://api.captain.dev"
API_KEY = "your_api_key"
COLLECTION_NAME = "customer_forms"

response = requests.post(
    f"{BASE_URL}/v3/collections/{COLLECTION_NAME}/evals",
    headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
        "Idempotency-Key": "eval-payments-2026-09-12-a",
    },
    json={
        "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
        "environment": "development",
        "configs": [
            {"name": "baseline"},
            {
                "name": "rerank-deep",
                "limit": 10,
                "semantic_ratio": 0.35,
                "rerank": {"enabled": True, "candidate_limit": 40, "model": "voyage-rerank-2.5"},
                "filter": {"review_status": {"$in": ["approved", "in_review"]}},
                "exclude_chunk_types": ["page_header", "page_footer", "footnote"],
                "max_chunks_per_document": 2,
            },
        ],
    },
    timeout=60.0,
)

data = response.json()
print(data["eval_id"], data["status"], data["preview"])

```

```typescript TypeScript REST
const BASE_URL = "https://api.captain.dev";
const API_KEY = "your_api_key";
const COLLECTION_NAME = "customer_forms";

const response = await fetch(
  `${BASE_URL}/v3/collections/${COLLECTION_NAME}/evals`,
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${API_KEY}`,
      "Content-Type": "application/json",
      "Idempotency-Key": "eval-payments-2026-09-12-a",
    },
    body: JSON.stringify({
      upload_id: "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
      environment: "development",
      configs: [
        { name: "baseline" },
        {
          name: "rerank-deep",
          limit: 10,
          semantic_ratio: 0.35,
          rerank: { enabled: true, candidate_limit: 40, model: "voyage-rerank-2.5" },
          filter: { review_status: { $in: ["approved", "in_review"] } },
          exclude_chunk_types: ["page_header", "page_footer", "footnote"],
          max_chunks_per_document: 2,
        },
      ],
    }),
  },
);

const data = await response.json();
console.log(data.eval_id, data.status, data.preview);

```