Documentation
¶
Overview ¶
Package adapters provides pluggable recording format adapters for Arena evaluation. It supports loading saved conversations from various formats (session recordings, arena output files, transcripts) into Arena-friendly structures.
Index ¶
- type ArenaOutputAdapter
- type MediaSource
- type ProviderSpec
- type RecordedEval
- type RecordedTransition
- type RecordingAdapter
- type RecordingMetadata
- type RecordingReference
- type Registry
- func (r *Registry) Enumerate(source, typeHint string) ([]RecordingReference, error)
- func (r *Registry) FindAdapter(path, typeHint string) RecordingAdapter
- func (r *Registry) Load(ref RecordingReference) ([]types.Message, *RecordingMetadata, error)
- func (r *Registry) Register(adapter RecordingAdapter)
- type SessionRecordingAdapter
- type TranscriptAdapter
- type TranscriptContentPart
- type TranscriptFile
- type TranscriptMediaPart
- type TranscriptMessage
- type TranscriptMetadata
- type TranscriptProviderSpec
- type TranscriptToolCall
- type TranscriptToolCallFunction
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ArenaOutputAdapter ¶
type ArenaOutputAdapter struct{}
ArenaOutputAdapter loads the per-run JSON that `promptarena run` writes into its output directory, one file per scenario/provider combination. That file is engine.RunResult marshalled: the run's identity, its full message list as PromptKit messages (tool calls and results inline), and both levels of assertion result — conversation-level at the top, per-turn on each assistant message under meta.assertions.
func NewArenaOutputAdapter ¶
func NewArenaOutputAdapter() *ArenaOutputAdapter
NewArenaOutputAdapter creates a new arena output adapter.
func (*ArenaOutputAdapter) CanHandle ¶
func (a *ArenaOutputAdapter) CanHandle(source, typeHint string) bool
CanHandle returns true for .json files that are not PromptKit session recordings, or for an "arena_output" type hint. Run output has no special extension — it is whatever `promptarena run` wrote — so the check is by content at Load time; here only the session-recording suffix is excluded.
func (*ArenaOutputAdapter) Enumerate ¶
func (a *ArenaOutputAdapter) Enumerate(source string) ([]RecordingReference, error)
Enumerate expands a source into individual recording references. For file-based sources, this expands glob patterns to matching files.
func (*ArenaOutputAdapter) Load ¶
func (a *ArenaOutputAdapter) Load(ref RecordingReference) ([]types.Message, *RecordingMetadata, error)
Load reads a run output file and returns its messages and metadata.
type MediaSource ¶
type MediaSource struct {
MIMEType string
Data string
URI string
Path string
Size int64
Width int
Height int
Duration int64 // milliseconds
}
MediaSource defines the source data for media content conversion.
type ProviderSpec ¶
type ProviderSpec struct {
Type string `json:"type" yaml:"type"`
Model string `json:"model" yaml:"model"`
ID string `json:"id" yaml:"id"`
}
ProviderSpec describes a provider configuration for judge targets.
type RecordedEval ¶
type RecordedEval struct {
ID string `json:"id,omitempty" yaml:"id,omitempty"`
Type string `json:"type" yaml:"type"`
Kind string `json:"kind,omitempty" yaml:"kind,omitempty"`
Score *float64 `json:"score,omitempty" yaml:"score,omitempty"`
Passed *bool `json:"passed,omitempty" yaml:"passed,omitempty"`
Message string `json:"message,omitempty" yaml:"message,omitempty"`
Params map[string]interface{} `json:"params,omitempty" yaml:"params,omitempty"`
Details map[string]interface{} `json:"details,omitempty" yaml:"details,omitempty"`
}
RecordedEval is one eval observation as a recording carries it, kept free of the assertions and evals packages so those can depend on this one.
Kind mirrors events.EvalKind: "eval" measures (Score set, Passed nil); "assertion" and "guardrail" carry a verdict in Passed. Params are the eval's original configuration when the recording kept it (arena run output does, under each result's "config"); they let a regression scenario assert the same thing rather than a guess at it.
type RecordedTransition ¶
type RecordedTransition struct {
From string `json:"from" yaml:"from"`
To string `json:"to" yaml:"to"`
Event string `json:"event,omitempty" yaml:"event,omitempty"`
PromptTask string `json:"prompt_task,omitempty" yaml:"prompt_task,omitempty"`
MessageIndex int `json:"message_index" yaml:"message_index"`
}
RecordedTransition is one workflow state change, placed by the index of the last message recorded before it (-1 when it preceded every message).
type RecordingAdapter ¶
type RecordingAdapter interface {
// CanHandle returns true if this adapter supports the given source/type hint.
// The source could be a file path, glob pattern, database query, etc.
// typeHint is an optional explicit format indicator from the eval config.
CanHandle(source, typeHint string) bool
// Enumerate expands a source into individual recording references.
// For file-based adapters, this expands glob patterns to matching files.
// For database adapters, this could execute a query and return record IDs.
// Returns a single-element slice for non-expandable sources.
Enumerate(source string) ([]RecordingReference, error)
// Load converts a recording to Arena message format.
// The reference should have been obtained from Enumerate.
// Returns the messages, metadata, and any error encountered.
Load(ref RecordingReference) ([]types.Message, *RecordingMetadata, error)
}
RecordingAdapter converts saved conversations from various formats into Arena-friendly structures for evaluation.
type RecordingMetadata ¶
type RecordingMetadata struct {
// JudgeTargets maps judge names to provider specifications.
// Used by LLM judge assertions to determine which provider to use.
JudgeTargets map[string]ProviderSpec `json:"judge_targets,omitempty" yaml:"judge_targets,omitempty"`
// ProviderInfo contains information about the original provider(s)
// that generated the recorded conversation.
ProviderInfo map[string]interface{} `json:"provider_info,omitempty" yaml:"provider_info,omitempty"`
// Tags are optional labels for categorizing/filtering recordings.
Tags []string `json:"tags,omitempty" yaml:"tags,omitempty"`
// Timestamps contains the timestamp for each turn in the conversation.
// The length should match the number of messages.
Timestamps []time.Time `json:"timestamps,omitempty" yaml:"timestamps,omitempty"`
// SessionID is the unique identifier for the recorded session.
SessionID string `json:"session_id,omitempty" yaml:"session_id,omitempty"`
// Duration is the total duration of the conversation.
Duration time.Duration `json:"duration,omitempty" yaml:"duration,omitempty"`
// Extras holds any additional metadata from the recording.
Extras map[string]interface{} `json:"extras,omitempty" yaml:"extras,omitempty"`
// ConversationAssertions are the conversation-level judged results the
// recording carries (kind "assertion" or "guardrail"). Nil when the source
// ran none.
ConversationAssertions []RecordedEval `json:"conversation_assertions,omitempty" yaml:"conversation_assertions,omitempty"` //nolint:lll
// TurnAssertions are per-turn judged results keyed by the index of the
// message they were evaluated against (an assistant message, for arena
// run output). Nil when the source carries none.
TurnAssertions map[int][]RecordedEval `json:"turn_assertions,omitempty" yaml:"turn_assertions,omitempty"`
// EvalResults are the pack-level eval observations the session produced:
// measurements (kind "eval") with a score and no verdict, and any
// guardrail outcomes recorded at session level. Conversation-level. Nil
// when the source carries none.
EvalResults []RecordedEval `json:"eval_results,omitempty" yaml:"eval_results,omitempty"`
// WorkflowTransitions are the recorded workflow state changes, in order.
// Nil when the session ran no workflow or the source does not record them.
WorkflowTransitions []RecordedTransition `json:"workflow_transitions,omitempty" yaml:"workflow_transitions,omitempty"`
}
RecordingMetadata contains metadata extracted from the recording that should flow through to the evaluation context.
type RecordingReference ¶
type RecordingReference struct {
// ID is a unique identifier for this recording reference.
// For file-based adapters, this is typically the file path.
// For database adapters, this could be a record ID.
ID string `json:"id" yaml:"id"`
// Source is the original source pattern/query that produced this reference.
// For file-based adapters, this is the original glob pattern or path.
Source string `json:"source" yaml:"source"`
// TypeHint is an optional format indicator (e.g., "session", "arena_output", "transcript").
TypeHint string `json:"type_hint,omitempty" yaml:"type_hint,omitempty"`
// Metadata contains adapter-specific metadata about this reference.
// This can include file size, modification time, database metadata, etc.
Metadata map[string]interface{} `json:"metadata,omitempty" yaml:"metadata,omitempty"`
}
RecordingReference is an opaque reference to a single recording. It abstracts the underlying storage mechanism (file, database, API, etc.) allowing adapters to enumerate and load recordings from various sources.
func EnumerateFiles ¶
func EnumerateFiles(source, typeHint string) ([]RecordingReference, error)
EnumerateFiles is a helper for file-based adapters to expand glob patterns. It returns recording references for each matching file. If the source doesn't contain glob characters, it returns a single reference.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry manages registered recording adapters.
func NewEmptyRegistry ¶
func NewEmptyRegistry() *Registry
NewEmptyRegistry creates an empty adapter registry without any built-in adapters. This is useful for testing or when you want to register only specific adapters.
func NewRegistry ¶
func NewRegistry() *Registry
NewRegistry creates a new adapter registry with default adapters registered.
func (*Registry) Enumerate ¶
func (r *Registry) Enumerate(source, typeHint string) ([]RecordingReference, error)
Enumerate expands a source into individual recording references. Uses the first adapter that can handle the source. Returns an error if no adapter can handle the source or if enumeration fails.
func (*Registry) FindAdapter ¶
func (r *Registry) FindAdapter(path, typeHint string) RecordingAdapter
FindAdapter returns the first adapter that can handle the given path and type hint. Returns nil if no adapter can handle the format.
func (*Registry) Load ¶
func (r *Registry) Load(ref RecordingReference) ([]types.Message, *RecordingMetadata, error)
Load finds an appropriate adapter and loads the recording from a reference. Returns an error if no adapter can handle the format or if loading fails.
func (*Registry) Register ¶
func (r *Registry) Register(adapter RecordingAdapter)
Register adds an adapter to the registry. Adapters are checked in registration order, so register more specific adapters before generic ones.
type SessionRecordingAdapter ¶
type SessionRecordingAdapter struct{}
SessionRecordingAdapter loads PromptKit session recordings: the artifact the recording middleware produces through the event store, either exported as a *.recording.json SessionRecording or left as the store's per-session JSONL. Parsing is delegated to recording.Load, which understands every shape the runtime writes, so this adapter cannot drift from the writer again.
func NewSessionRecordingAdapter ¶
func NewSessionRecordingAdapter() *SessionRecordingAdapter
NewSessionRecordingAdapter creates a new session recording adapter.
func (*SessionRecordingAdapter) CanHandle ¶
func (a *SessionRecordingAdapter) CanHandle(source, typeHint string) bool
CanHandle returns true for *.recording.json or *.jsonl files, or a "session" type hint.
func (*SessionRecordingAdapter) Enumerate ¶
func (a *SessionRecordingAdapter) Enumerate(source string) ([]RecordingReference, error)
Enumerate expands a source into individual recording references. For file-based sources, this expands glob patterns to matching files.
func (*SessionRecordingAdapter) Load ¶
func (a *SessionRecordingAdapter) Load(ref RecordingReference) ([]types.Message, *RecordingMetadata, error)
Load reads a session recording and returns its messages: one per message.created event, in order. Every other event type is skipped.
type TranscriptAdapter ¶
type TranscriptAdapter struct{}
TranscriptAdapter loads transcript YAML files (*.transcript.yaml).
func NewTranscriptAdapter ¶
func NewTranscriptAdapter() *TranscriptAdapter
NewTranscriptAdapter creates a new transcript adapter.
func (*TranscriptAdapter) CanHandle ¶
func (a *TranscriptAdapter) CanHandle(source, typeHint string) bool
CanHandle returns true for *.transcript.yaml files or "transcript" type hint.
func (*TranscriptAdapter) Enumerate ¶
func (a *TranscriptAdapter) Enumerate(source string) ([]RecordingReference, error)
Enumerate expands a source into individual recording references. For file-based sources, this expands glob patterns to matching files.
func (*TranscriptAdapter) Load ¶
func (a *TranscriptAdapter) Load(ref RecordingReference) ([]types.Message, *RecordingMetadata, error)
Load reads a transcript file and converts it to Arena messages.
type TranscriptContentPart ¶
type TranscriptContentPart struct {
Type string `yaml:"type"`
Text *string `yaml:"text,omitempty"`
Media *TranscriptMediaPart `yaml:"media,omitempty"`
}
TranscriptContentPart represents a content part in the transcript.
type TranscriptFile ¶
type TranscriptFile struct {
Metadata TranscriptMetadata `yaml:"metadata"`
Messages []TranscriptMessage `yaml:"messages"`
}
TranscriptFile represents the structure of a *.transcript.yaml file.
type TranscriptMediaPart ¶
type TranscriptMediaPart struct {
MIMEType string `yaml:"mime_type,omitempty"`
Data string `yaml:"data,omitempty"` // Base64 encoded
URI string `yaml:"uri,omitempty"` // URL
Path string `yaml:"path,omitempty"` // File path
Size int64 `yaml:"size,omitempty"`
Width int `yaml:"width,omitempty"`
Height int `yaml:"height,omitempty"`
Duration int64 `yaml:"duration,omitempty"` // milliseconds
}
TranscriptMediaPart represents media content in the transcript.
type TranscriptMessage ¶
type TranscriptMessage struct {
Role string `yaml:"role"`
Content string `yaml:"content"`
Name string `yaml:"name,omitempty"`
Timestamp string `yaml:"timestamp,omitempty"`
Parts []TranscriptContentPart `yaml:"parts,omitempty"`
ToolCalls []TranscriptToolCall `yaml:"tool_calls,omitempty"`
ToolCallID string `yaml:"tool_call_id,omitempty"`
}
TranscriptMessage represents a message in the transcript.
type TranscriptMetadata ¶
type TranscriptMetadata struct {
SessionID string `yaml:"session_id,omitempty"`
Provider string `yaml:"provider,omitempty"`
Model string `yaml:"model,omitempty"`
Tags []string `yaml:"tags,omitempty"`
JudgeTargets map[string]TranscriptProviderSpec `yaml:"judge_targets,omitempty"`
}
TranscriptMetadata contains metadata about the transcript.
type TranscriptProviderSpec ¶
type TranscriptProviderSpec struct {
Type string `yaml:"type"`
Model string `yaml:"model"`
ID string `yaml:"id"`
}
TranscriptProviderSpec describes a provider for judge targets in transcripts.
type TranscriptToolCall ¶
type TranscriptToolCall struct {
ID string `yaml:"id"`
Type string `yaml:"type"`
Function TranscriptToolCallFunction `yaml:"function"`
}
TranscriptToolCall represents a tool call in the transcript.
type TranscriptToolCallFunction ¶
type TranscriptToolCallFunction struct {
Name string `yaml:"name"`
Arguments string `yaml:"arguments"`
}
TranscriptToolCallFunction represents a function call within a tool call.