Help us shape the future of Go! Take the 2026 Go Developer Survey

adapters

package
v2.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 30, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package adapters provides pluggable recording format adapters for Arena evaluation. It supports loading saved conversations from various formats (session recordings, arena output files, transcripts) into Arena-friendly structures.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ArenaOutputAdapter

type ArenaOutputAdapter struct{}

ArenaOutputAdapter loads the per-run JSON that `promptarena run` writes into its output directory, one file per scenario/provider combination. That file is engine.RunResult marshalled: the run's identity, its full message list as PromptKit messages (tool calls and results inline), and both levels of assertion result — conversation-level at the top, per-turn on each assistant message under meta.assertions.

func NewArenaOutputAdapter

func NewArenaOutputAdapter() *ArenaOutputAdapter

NewArenaOutputAdapter creates a new arena output adapter.

func (*ArenaOutputAdapter) CanHandle

func (a *ArenaOutputAdapter) CanHandle(source, typeHint string) bool

CanHandle returns true for .json files that are not PromptKit session recordings, or for an "arena_output" type hint. Run output has no special extension — it is whatever `promptarena run` wrote — so the check is by content at Load time; here only the session-recording suffix is excluded.

func (*ArenaOutputAdapter) Enumerate

func (a *ArenaOutputAdapter) Enumerate(source string) ([]RecordingReference, error)

Enumerate expands a source into individual recording references. For file-based sources, this expands glob patterns to matching files.

func (*ArenaOutputAdapter) Load

Load reads a run output file and returns its messages and metadata.

type MediaSource

type MediaSource struct {
	MIMEType string
	Data     string
	URI      string
	Path     string
	Size     int64
	Width    int
	Height   int
	Duration int64 // milliseconds
}

MediaSource defines the source data for media content conversion.

type ProviderSpec

type ProviderSpec struct {
	Type  string `json:"type" yaml:"type"`
	Model string `json:"model" yaml:"model"`
	ID    string `json:"id" yaml:"id"`
}

ProviderSpec describes a provider configuration for judge targets.

type RecordedEval

type RecordedEval struct {
	ID      string                 `json:"id,omitempty" yaml:"id,omitempty"`
	Type    string                 `json:"type" yaml:"type"`
	Kind    string                 `json:"kind,omitempty" yaml:"kind,omitempty"`
	Score   *float64               `json:"score,omitempty" yaml:"score,omitempty"`
	Passed  *bool                  `json:"passed,omitempty" yaml:"passed,omitempty"`
	Message string                 `json:"message,omitempty" yaml:"message,omitempty"`
	Params  map[string]interface{} `json:"params,omitempty" yaml:"params,omitempty"`
	Details map[string]interface{} `json:"details,omitempty" yaml:"details,omitempty"`
}

RecordedEval is one eval observation as a recording carries it, kept free of the assertions and evals packages so those can depend on this one.

Kind mirrors events.EvalKind: "eval" measures (Score set, Passed nil); "assertion" and "guardrail" carry a verdict in Passed. Params are the eval's original configuration when the recording kept it (arena run output does, under each result's "config"); they let a regression scenario assert the same thing rather than a guess at it.

type RecordedTransition

type RecordedTransition struct {
	From         string `json:"from" yaml:"from"`
	To           string `json:"to" yaml:"to"`
	Event        string `json:"event,omitempty" yaml:"event,omitempty"`
	PromptTask   string `json:"prompt_task,omitempty" yaml:"prompt_task,omitempty"`
	MessageIndex int    `json:"message_index" yaml:"message_index"`
}

RecordedTransition is one workflow state change, placed by the index of the last message recorded before it (-1 when it preceded every message).

type RecordingAdapter

type RecordingAdapter interface {
	// CanHandle returns true if this adapter supports the given source/type hint.
	// The source could be a file path, glob pattern, database query, etc.
	// typeHint is an optional explicit format indicator from the eval config.
	CanHandle(source, typeHint string) bool

	// Enumerate expands a source into individual recording references.
	// For file-based adapters, this expands glob patterns to matching files.
	// For database adapters, this could execute a query and return record IDs.
	// Returns a single-element slice for non-expandable sources.
	Enumerate(source string) ([]RecordingReference, error)

	// Load converts a recording to Arena message format.
	// The reference should have been obtained from Enumerate.
	// Returns the messages, metadata, and any error encountered.
	Load(ref RecordingReference) ([]types.Message, *RecordingMetadata, error)
}

RecordingAdapter converts saved conversations from various formats into Arena-friendly structures for evaluation.

type RecordingMetadata

type RecordingMetadata struct {
	// JudgeTargets maps judge names to provider specifications.
	// Used by LLM judge assertions to determine which provider to use.
	JudgeTargets map[string]ProviderSpec `json:"judge_targets,omitempty" yaml:"judge_targets,omitempty"`

	// ProviderInfo contains information about the original provider(s)
	// that generated the recorded conversation.
	ProviderInfo map[string]interface{} `json:"provider_info,omitempty" yaml:"provider_info,omitempty"`

	// Tags are optional labels for categorizing/filtering recordings.
	Tags []string `json:"tags,omitempty" yaml:"tags,omitempty"`

	// Timestamps contains the timestamp for each turn in the conversation.
	// The length should match the number of messages.
	Timestamps []time.Time `json:"timestamps,omitempty" yaml:"timestamps,omitempty"`

	// SessionID is the unique identifier for the recorded session.
	SessionID string `json:"session_id,omitempty" yaml:"session_id,omitempty"`

	// Duration is the total duration of the conversation.
	Duration time.Duration `json:"duration,omitempty" yaml:"duration,omitempty"`

	// Extras holds any additional metadata from the recording.
	Extras map[string]interface{} `json:"extras,omitempty" yaml:"extras,omitempty"`

	// ConversationAssertions are the conversation-level judged results the
	// recording carries (kind "assertion" or "guardrail"). Nil when the source
	// ran none.
	ConversationAssertions []RecordedEval `json:"conversation_assertions,omitempty" yaml:"conversation_assertions,omitempty"` //nolint:lll

	// TurnAssertions are per-turn judged results keyed by the index of the
	// message they were evaluated against (an assistant message, for arena
	// run output). Nil when the source carries none.
	TurnAssertions map[int][]RecordedEval `json:"turn_assertions,omitempty" yaml:"turn_assertions,omitempty"`

	// EvalResults are the pack-level eval observations the session produced:
	// measurements (kind "eval") with a score and no verdict, and any
	// guardrail outcomes recorded at session level. Conversation-level. Nil
	// when the source carries none.
	EvalResults []RecordedEval `json:"eval_results,omitempty" yaml:"eval_results,omitempty"`

	// WorkflowTransitions are the recorded workflow state changes, in order.
	// Nil when the session ran no workflow or the source does not record them.
	WorkflowTransitions []RecordedTransition `json:"workflow_transitions,omitempty" yaml:"workflow_transitions,omitempty"`
}

RecordingMetadata contains metadata extracted from the recording that should flow through to the evaluation context.

type RecordingReference

type RecordingReference struct {
	// ID is a unique identifier for this recording reference.
	// For file-based adapters, this is typically the file path.
	// For database adapters, this could be a record ID.
	ID string `json:"id" yaml:"id"`

	// Source is the original source pattern/query that produced this reference.
	// For file-based adapters, this is the original glob pattern or path.
	Source string `json:"source" yaml:"source"`

	// TypeHint is an optional format indicator (e.g., "session", "arena_output", "transcript").
	TypeHint string `json:"type_hint,omitempty" yaml:"type_hint,omitempty"`

	// Metadata contains adapter-specific metadata about this reference.
	// This can include file size, modification time, database metadata, etc.
	Metadata map[string]interface{} `json:"metadata,omitempty" yaml:"metadata,omitempty"`
}

RecordingReference is an opaque reference to a single recording. It abstracts the underlying storage mechanism (file, database, API, etc.) allowing adapters to enumerate and load recordings from various sources.

func EnumerateFiles

func EnumerateFiles(source, typeHint string) ([]RecordingReference, error)

EnumerateFiles is a helper for file-based adapters to expand glob patterns. It returns recording references for each matching file. If the source doesn't contain glob characters, it returns a single reference.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry manages registered recording adapters.

func NewEmptyRegistry

func NewEmptyRegistry() *Registry

NewEmptyRegistry creates an empty adapter registry without any built-in adapters. This is useful for testing or when you want to register only specific adapters.

func NewRegistry

func NewRegistry() *Registry

NewRegistry creates a new adapter registry with default adapters registered.

func (*Registry) Enumerate

func (r *Registry) Enumerate(source, typeHint string) ([]RecordingReference, error)

Enumerate expands a source into individual recording references. Uses the first adapter that can handle the source. Returns an error if no adapter can handle the source or if enumeration fails.

func (*Registry) FindAdapter

func (r *Registry) FindAdapter(path, typeHint string) RecordingAdapter

FindAdapter returns the first adapter that can handle the given path and type hint. Returns nil if no adapter can handle the format.

func (*Registry) Load

Load finds an appropriate adapter and loads the recording from a reference. Returns an error if no adapter can handle the format or if loading fails.

func (*Registry) Register

func (r *Registry) Register(adapter RecordingAdapter)

Register adds an adapter to the registry. Adapters are checked in registration order, so register more specific adapters before generic ones.

type SessionRecordingAdapter

type SessionRecordingAdapter struct{}

SessionRecordingAdapter loads PromptKit session recordings: the artifact the recording middleware produces through the event store, either exported as a *.recording.json SessionRecording or left as the store's per-session JSONL. Parsing is delegated to recording.Load, which understands every shape the runtime writes, so this adapter cannot drift from the writer again.

func NewSessionRecordingAdapter

func NewSessionRecordingAdapter() *SessionRecordingAdapter

NewSessionRecordingAdapter creates a new session recording adapter.

func (*SessionRecordingAdapter) CanHandle

func (a *SessionRecordingAdapter) CanHandle(source, typeHint string) bool

CanHandle returns true for *.recording.json or *.jsonl files, or a "session" type hint.

func (*SessionRecordingAdapter) Enumerate

func (a *SessionRecordingAdapter) Enumerate(source string) ([]RecordingReference, error)

Enumerate expands a source into individual recording references. For file-based sources, this expands glob patterns to matching files.

func (*SessionRecordingAdapter) Load

Load reads a session recording and returns its messages: one per message.created event, in order. Every other event type is skipped.

type TranscriptAdapter

type TranscriptAdapter struct{}

TranscriptAdapter loads transcript YAML files (*.transcript.yaml).

func NewTranscriptAdapter

func NewTranscriptAdapter() *TranscriptAdapter

NewTranscriptAdapter creates a new transcript adapter.

func (*TranscriptAdapter) CanHandle

func (a *TranscriptAdapter) CanHandle(source, typeHint string) bool

CanHandle returns true for *.transcript.yaml files or "transcript" type hint.

func (*TranscriptAdapter) Enumerate

func (a *TranscriptAdapter) Enumerate(source string) ([]RecordingReference, error)

Enumerate expands a source into individual recording references. For file-based sources, this expands glob patterns to matching files.

func (*TranscriptAdapter) Load

Load reads a transcript file and converts it to Arena messages.

type TranscriptContentPart

type TranscriptContentPart struct {
	Type  string               `yaml:"type"`
	Text  *string              `yaml:"text,omitempty"`
	Media *TranscriptMediaPart `yaml:"media,omitempty"`
}

TranscriptContentPart represents a content part in the transcript.

type TranscriptFile

type TranscriptFile struct {
	Metadata TranscriptMetadata  `yaml:"metadata"`
	Messages []TranscriptMessage `yaml:"messages"`
}

TranscriptFile represents the structure of a *.transcript.yaml file.

type TranscriptMediaPart

type TranscriptMediaPart struct {
	MIMEType string `yaml:"mime_type,omitempty"`
	Data     string `yaml:"data,omitempty"` // Base64 encoded
	URI      string `yaml:"uri,omitempty"`  // URL
	Path     string `yaml:"path,omitempty"` // File path
	Size     int64  `yaml:"size,omitempty"`
	Width    int    `yaml:"width,omitempty"`
	Height   int    `yaml:"height,omitempty"`
	Duration int64  `yaml:"duration,omitempty"` // milliseconds
}

TranscriptMediaPart represents media content in the transcript.

type TranscriptMessage

type TranscriptMessage struct {
	Role       string                  `yaml:"role"`
	Content    string                  `yaml:"content"`
	Name       string                  `yaml:"name,omitempty"`
	Timestamp  string                  `yaml:"timestamp,omitempty"`
	Parts      []TranscriptContentPart `yaml:"parts,omitempty"`
	ToolCalls  []TranscriptToolCall    `yaml:"tool_calls,omitempty"`
	ToolCallID string                  `yaml:"tool_call_id,omitempty"`
}

TranscriptMessage represents a message in the transcript.

type TranscriptMetadata

type TranscriptMetadata struct {
	SessionID    string                            `yaml:"session_id,omitempty"`
	Provider     string                            `yaml:"provider,omitempty"`
	Model        string                            `yaml:"model,omitempty"`
	Tags         []string                          `yaml:"tags,omitempty"`
	JudgeTargets map[string]TranscriptProviderSpec `yaml:"judge_targets,omitempty"`
}

TranscriptMetadata contains metadata about the transcript.

type TranscriptProviderSpec

type TranscriptProviderSpec struct {
	Type  string `yaml:"type"`
	Model string `yaml:"model"`
	ID    string `yaml:"id"`
}

TranscriptProviderSpec describes a provider for judge targets in transcripts.

type TranscriptToolCall

type TranscriptToolCall struct {
	ID       string                     `yaml:"id"`
	Type     string                     `yaml:"type"`
	Function TranscriptToolCallFunction `yaml:"function"`
}

TranscriptToolCall represents a tool call in the transcript.

type TranscriptToolCallFunction

type TranscriptToolCallFunction struct {
	Name      string `yaml:"name"`
	Arguments string `yaml:"arguments"`
}

TranscriptToolCallFunction represents a function call within a tool call.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL