Queue Evaluation

import requests
BASE_URL = "https://api.captain.dev"
API_KEY = "your_api_key"
COLLECTION_NAME = "customer_forms"
response = requests.post(
    f"{BASE_URL}/v3/collections/{COLLECTION_NAME}/evals",
    headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
        "Idempotency-Key": "eval-payments-2026-09-12-a",
    },
    json={
        "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
        "environment": "development",
        "configs": [
            {"name": "baseline"},
            {
                "name": "rerank-deep",
                "limit": 10,
                "semantic_ratio": 0.35,
                "rerank": {"enabled": True, "candidate_limit": 40, "model": "voyage-rerank-2.5"},
                "filter": {"review_status": {"$in": ["approved", "in_review"]}},
                "exclude_chunk_types": ["page_header", "page_footer", "footnote"],
                "max_chunks_per_document": 2,
            },
        ],
    },
    timeout=60.0,
)
data = response.json()
print(data["eval_id"], data["status"], data["preview"])

{
  "collection_name": "customer_forms",
  "configs": [
    {
      "explicit_fields": [],
      "name": "baseline",
      "query": {
        "limit": 10,
        "rerank": {
          "enabled": false,
          "candidate_limit": null,
          "model": null
        },
        "semantic_ratio": 0.5,
        "boost": [],
        "exclude_chunk_types": [],
        "filter": null,
        "include": {
          "document": true,
          "metadata": true,
          "regions": false,
          "relations": false,
          "document_metadata": false,
          "related_chunks": false,
          "neighboring_chunks": false,
          "archived": false
        },
        "max_chunks_per_document": null,
        "relation_direction": "outgoing",
        "relation_types": null
      }
    },
    {
      "explicit_fields": [
        "exclude_chunk_types",
        "filter",
        "limit",
        "max_chunks_per_document",
        "rerank",
        "semantic_ratio"
      ],
      "name": "rerank-deep",
      "query": {
        "limit": 10,
        "rerank": {
          "enabled": true,
          "candidate_limit": 40,
          "model": "voyage-rerank-2.5"
        },
        "semantic_ratio": 0.35,
        "boost": [],
        "exclude_chunk_types": [
          "page_header",
          "page_footer",
          "footnote"
        ],
        "filter": {
          "review_status": {
            "$in": [
              "approved",
              "in_review"
            ]
          }
        },
        "include": {
          "document": true,
          "metadata": true,
          "regions": false,
          "relations": false,
          "document_metadata": false,
          "related_chunks": false,
          "neighboring_chunks": false,
          "archived": false
        },
        "max_chunks_per_document": 2,
        "relation_direction": "outgoing",
        "relation_types": null
      }
    }
  ],
  "created_at": "2026-09-12T17:30:01Z",
  "eval_id": "eval_019b4c2f11d04c8e9a7b3d5c2e1840ff",
  "idempotency_key": "eval-payments-2026-09-12-a",
  "items_page": {
    "limit": 100,
    "total": 3,
    "next_cursor": null
  },
  "preview": {
    "billable_units": 6,
    "cases": 3,
    "cases_error": 0,
    "configs": 2,
    "units": 6
  },
  "progress": {
    "cases_error": 0,
    "cases_total": 3,
    "configs_failed": 0,
    "configs_total": 2,
    "elapsed_seconds": 0,
    "percent": 0,
    "units_completed": 0,
    "units_error": 0,
    "units_failed": 0,
    "units_total": 6,
    "eta_seconds": null,
    "qps": null
  },
  "request_id": "req_20260912_eval001",
  "status": "pending",
  "updated_at": "2026-09-12T17:30:01Z",
  "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
  "billing": null,
  "completed_at": null,
  "environment": "development",
  "error_code": null,
  "error_message": null,
  "items": [],
  "scorecards": {},
  "started_at": null
}

Queue an evaluation of an uploaded case set against one to eight named query configurations. Idempotency-Key is required: within 24 hours the same key with the same body returns the stored 201 body with status 200; the same key with a different body is 409 (IDEMPOTENCY_KEY_REUSE). Gold is resolved at accept time (document id, then unique basename); unresolved cases are error and never queried or billed. Each configs[] entry is the v3 query body minus query and is validated the same way. Each unit (one question under one config) is billed as one query, so whether it costs credits depends on the plan; only units that ran (scored, or timed out at the 30 s limit) are counted. Scoring, limits, billing and the NDJSON format are on the Evaluations guide.

Path parameters

collection_namestringRequired

Request

This endpoint expects an object.

configslist of objectsRequired

upload_idstringRequired5-64 characters

environmentstring or nullOptional<=64 characters

Echo-only label for the scorecard. Does not switch collections.

Response

Idempotent replay: the stored original 201 body (including the original eval_id), not live status. Poll Get Eval for progress.

collection_namestring

configslist of objects

created_atstring

eval_idstring

idempotency_keystring

items_pageobject

previewobject

progressobject

request_idstring

statusenum

Allowed values:

updated_atstring

upload_idstring

billingobject or nullOptional

completed_atstring or nullOptional

environmentstring or nullOptional

error_codestring or nullOptional

error_messagestring or nullOptional

itemslist of objectsOptional

scorecardsmap from strings to objectsOptional

started_atstring or nullOptional

Errors

400

Bad Request Error

401

Unauthorized Error

403

Forbidden Error

404

Not Found Error

409

Conflict Error

429

Too Many Requests Error

Path parameters

Headers

Request

Response

Errors