Documentation
¶
Index ¶
- func Encode(b *MetadataBuilder, v Value) []byte
- func Marshal(v any) (metadata, value []byte, err error)
- func Unmarshal(metadata, value []byte) (any, error)
- type Array
- type BasicType
- type Builder
- func (b *Builder) BeginArray()
- func (b *Builder) BeginObject()
- func (b *Builder) Binary(v []byte)
- func (b *Builder) Bool(v bool)
- func (b *Builder) Bytes() ([]byte, error)
- func (b *Builder) Date(days int32)
- func (b *Builder) Decimal4(unscaled int32, scale byte)
- func (b *Builder) Decimal8(unscaled int64, scale byte)
- func (b *Builder) Decimal16(unscaled [16]byte, scale byte)
- func (b *Builder) Double(v float64)
- func (b *Builder) EndArray()
- func (b *Builder) EndObject()
- func (b *Builder) Err() error
- func (b *Builder) Field(name string)
- func (b *Builder) Finish() (metadata, value []byte, err error)
- func (b *Builder) Float(v float32)
- func (b *Builder) Int8(v int8)
- func (b *Builder) Int16(v int16)
- func (b *Builder) Int32(v int32)
- func (b *Builder) Int64(v int64)
- func (b *Builder) Null()
- func (b *Builder) Reset()
- func (b *Builder) String(v string)
- func (b *Builder) Time(micros int64)
- func (b *Builder) Timestamp(micros int64)
- func (b *Builder) TimestampNTZ(micros int64)
- func (b *Builder) TimestampNTZNanos(nanos int64)
- func (b *Builder) TimestampNanos(nanos int64)
- func (b *Builder) UUID(v uuid.UUID)
- type Field
- type Loc
- type Metadata
- type MetadataBuilder
- type Object
- type PrimitiveType
- type Value
- func Binary(v []byte) Value
- func Bool(v bool) Value
- func Date(v int32) Value
- func Decimal4(v int32, scale byte) Value
- func Decimal8(v int64, scale byte) Value
- func Decimal16(v [16]byte, scale byte) Value
- func Decode(m Metadata, data []byte) (Value, error)
- func Double(v float64) Value
- func Float(v float32) Value
- func Int8(v int8) Value
- func Int16(v int16) Value
- func Int32(v int32) Value
- func Int64(v int64) Value
- func MakeArray(elements []Value) Value
- func MakeObject(fields []Field) Value
- func Null() Value
- func String(v string) Value
- func Time(v int64) Value
- func Timestamp(v int64) Value
- func TimestampNTZ(v int64) Value
- func TimestampNTZNanos(v int64) Value
- func TimestampNanos(v int64) Value
- func UUID(v uuid.UUID) Value
- func ValueOf(v any) (Value, error)
- func (v Value) ArrayValue() Array
- func (v Value) Basic() BasicType
- func (v Value) BoolValue() bool
- func (v Value) Bytes() []byte
- func (v Value) Decimal16Value() [16]byte
- func (v Value) Equal(other Value) bool
- func (v Value) FloatValue() float64
- func (v Value) GoValue() any
- func (v Value) Int() int64
- func (v Value) IsNull() bool
- func (v Value) ObjectValue() Object
- func (v Value) Scale() byte
- func (v Value) Str() string
- func (v Value) Type() PrimitiveType
- func (v Value) UUIDValue() uuid.UUID
- func (v Value) Write(w ValueBuilder)
- type ValueBuilder
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Encode ¶
func Encode(b *MetadataBuilder, v Value) []byte
Encode encodes a variant Value into its binary representation. Field names for objects are registered with the provided MetadataBuilder. Returns the encoded value bytes.
Types ¶
type BasicType ¶
type BasicType byte
BasicType represents the basic type encoded in bits 0-1 of a value header byte.
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder is a streaming encoder for variant values: a ValueBuilder that encodes events directly to the variant binary format in a single growing buffer, without building a Value tree.
Values are encoded as their events arrive. When a container closes, its header (field IDs, offsets) is spliced in front of the already-encoded children with one memmove, so encoding is a single pass with no per-value allocations beyond buffer growth. Object field values are kept in event order and only the header entries are sorted by field name, which the encoding permits (field IDs must be sorted, field offsets need not be monotonic).
The zero value is ready to use and owns its metadata dictionary; Finish returns both the metadata and the value. NewBuilderWithMetadata creates a builder that interns field names into a shared, caller-owned dictionary, which is how multiple values (e.g. all residuals of one shredded row) share one dictionary.
A Builder may be reused after Reset. Reset retains internal buffers, so a long-lived Builder encodes a stream of values with amortized zero allocation. A Builder must not be copied after first use.
func NewBuilderWithMetadata ¶
func NewBuilderWithMetadata(meta *MetadataBuilder) *Builder
NewBuilderWithMetadata returns a Builder that interns object field names into the given shared metadata dictionary instead of an internal one. Finish builds the shared dictionary; callers encoding multiple values against one dictionary typically use Bytes and build the metadata themselves once.
func (*Builder) BeginArray ¶
func (b *Builder) BeginArray()
func (*Builder) BeginObject ¶
func (b *Builder) BeginObject()
func (*Builder) Bytes ¶
Bytes returns the encoded value bytes without building the metadata. The returned slice aliases the builder's internal buffer and is invalidated by Reset or further writes.
func (*Builder) Finish ¶
Finish returns the encoded metadata and value. It fails if no value or an incomplete value has been written. The value slice aliases the builder's internal buffer and is invalidated by Reset or further writes, the same as Bytes; the metadata slice is freshly allocated.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset clears the builder for reuse, retaining internal buffers. If the builder owns its metadata dictionary the dictionary is cleared too; a shared dictionary (NewBuilderWithMetadata) is left untouched.
func (*Builder) TimestampNTZ ¶
func (*Builder) TimestampNTZNanos ¶
func (*Builder) TimestampNanos ¶
type Loc ¶
type Loc uint8
Loc describes where the value of a variant path lives for one row (or one list element) of a columnar reader window.
const ( // LocMissing means the path has no value: an enclosing level is null, // the field is absent from its parent object, or the whole variant // column is null for the row. LocMissing Loc = iota // LocNull means the path holds the variant null value. LocNull // LocTyped means the value is a scalar stored in a shredded typed_value // column; it appears in the cursor's typed vector. LocTyped // LocTypedObject means the value is an object stored (at least // partially) in shredded form. Leftover fields of a partially shredded // object are available through the residual. LocTypedObject // LocTypedList means the value is a list stored in shredded form. LocTypedList // LocResidual means the value exists only as variant binary at this // position. LocResidual )
type Metadata ¶
Metadata holds the decoded metadata dictionary for a variant value. Sorted reports whether the dictionary strings were stored in lexicographic order (the sorted_strings header bit).
func DecodeMetadata ¶
DecodeMetadata decodes a variant metadata binary blob.
Format: header(1) | dictionary_size(uint) | offsets(dictionary_size+1) | string_data
Header bits (per the spec's metadata encoding grammar, header = version | sorted_strings << 4 | offset_size_minus_one << 6):
0-3: version (must be 1) 4: sorted_strings 6-7: offset_size_minus_one
Like Decode, DecodeMetadata rejects what the spec declares invalid (non-UTF-8 dictionary strings, sizes exceeding the data) and does not validate the unused header bit, per VariantEncoding.md: "Bit 5 (marked R) is reserved; it must be ignored by readers."
type MetadataBuilder ¶
type MetadataBuilder struct {
// contains filtered or unexported fields
}
MetadataBuilder builds a variant metadata dictionary. Field names are interned into a single contiguous byte buffer so a high-cardinality key set pays for buffer growth rather than a heap string per key.
func (*MetadataBuilder) Add ¶
func (b *MetadataBuilder) Add(s string) int
Add interns s into the dictionary and returns its stable index. The string is copied into the builder's buffer, so callers may reuse the argument's backing memory after Add returns.
func (*MetadataBuilder) AppendTo ¶
func (b *MetadataBuilder) AppendTo(dst []byte) []byte
AppendTo appends the encoded binary representation of the dictionary to dst and returns the extended slice. The dictionary indices in the output match those returned by Add, so encoded values referencing those indices remain valid.
The variant format encodes dictionary offsets in at most 4 bytes, so the interned names must total at most math.MaxUint32 bytes; callers growing the dictionary without bound must check Size before encoding (as Builder.Finish and parquet.VariantColumnWriter do).
func (*MetadataBuilder) Build ¶
func (b *MetadataBuilder) Build() (Metadata, []byte)
Build returns the decoded Metadata and the encoded binary representation. The dictionary indices in the output match those returned by Add, so encoded values referencing those indices remain valid. The returned Metadata owns its strings and remains valid after Reset.
func (*MetadataBuilder) Reset ¶
func (b *MetadataBuilder) Reset()
Reset clears the builder for reuse, retaining the string buffer's capacity.
func (*MetadataBuilder) Size ¶
func (b *MetadataBuilder) Size() int
Size returns the total number of bytes of interned dictionary strings.
type Object ¶
type Object struct {
Fields []Field
}
Object represents a variant object (ordered set of named fields).
type PrimitiveType ¶
type PrimitiveType byte
PrimitiveType represents the primitive type encoded in bits 2-7 of a value header byte when BasicType is BasicPrimitive.
const ( PrimitiveNull PrimitiveType = 0 PrimitiveTrue PrimitiveType = 1 PrimitiveFalse PrimitiveType = 2 PrimitiveInt8 PrimitiveType = 3 PrimitiveInt16 PrimitiveType = 4 PrimitiveInt32 PrimitiveType = 5 PrimitiveInt64 PrimitiveType = 6 PrimitiveDouble PrimitiveType = 7 PrimitiveDecimal4 PrimitiveType = 8 PrimitiveDecimal8 PrimitiveType = 9 PrimitiveDecimal16 PrimitiveType = 10 PrimitiveDate PrimitiveType = 11 PrimitiveTimestamp PrimitiveType = 12 PrimitiveTimestampNTZ PrimitiveType = 13 PrimitiveFloat PrimitiveType = 14 PrimitiveBinary PrimitiveType = 15 PrimitiveString PrimitiveType = 16 PrimitiveTime PrimitiveType = 17 PrimitiveTimestampNanos PrimitiveType = 18 PrimitiveTimestampNTZNanos PrimitiveType = 19 PrimitiveUUID PrimitiveType = 20 )
type Value ¶
type Value struct {
// contains filtered or unexported fields
}
Value represents a variant value. The zero value is Null.
func Decimal4 ¶
Decimal4 returns a 4-byte decimal variant value with the given unscaled value and scale.
func Decimal8 ¶
Decimal8 returns an 8-byte decimal variant value with the given unscaled value and scale.
func Decimal16 ¶
Decimal16 returns a 16-byte decimal variant value with the given unscaled value and scale.
func Decode ¶
Decode decodes a variant value from its binary representation using the given metadata dictionary.
Decode rejects input the spec declares invalid: strings that are not valid UTF-8, objects with duplicate field names, and structures whose declared sizes exceed the data. It accepts input that is non-canonical but unambiguous, as the spec directs for readers: reserved header bits are ignored, object fields need not be listed in lexicographic order, and bytes past the end of the value are ignored (values are self-delimiting).
func MakeObject ¶
MakeObject creates an object variant value from a slice of fields.
func TimestampNTZ ¶
TimestampNTZ returns a timestamp without timezone variant value (microseconds since epoch).
func TimestampNTZNanos ¶
TimestampNTZNanos returns a timestamp without timezone variant value (nanoseconds since epoch).
func TimestampNanos ¶
TimestampNanos returns a timestamp variant value (nanoseconds since epoch, UTC).
func ValueOf ¶
ValueOf converts a Go value to a variant Value without encoding it. It is useful for callers that need to inspect or partially encode the value, such as shredded variant writers.
func (Value) ArrayValue ¶
ArrayValue returns the Array. Panics if not an array.
func (Value) Decimal16Value ¶
Decimal16Value returns the 16-byte decimal value.
func (Value) Equal ¶
Equal reports whether two variant values are structurally equal: the same value with the same primitive type. Object fields are compared without regard to order, short strings and primitive strings with the same content are considered equal, and floating-point values are compared bitwise (NaN equals NaN, negative zero does not equal positive zero).
func (Value) FloatValue ¶
FloatValue returns the floating-point value. Works for Float and Double.
func (Value) GoValue ¶
GoValue converts the variant value to a natural Go type.
Mapping:
Null → nil
True/False → bool
Int8 → int8, Int16 → int16, Int32 → int32, Int64 → int64
Float → float32, Double → float64
String → string, Binary → []byte
Date → int32 (days since epoch)
Timestamp/TimestampNanos → time.Time (UTC)
TimestampNTZ/TimestampNTZNanos → int64 (microseconds/nanoseconds since
epoch; these have no time zone, so no time.Time instant exists for
them)
Time → int64 (microseconds since midnight)
UUID → uuid.UUID
Decimal4 → int32 (unscaled), Decimal8 → int64 (unscaled), Decimal16 → [16]byte
Object → map[string]any
Array → []any
func (Value) Int ¶
Int returns the integer value as int64. Works for Int8, Int16, Int32, Int64, Date, Timestamp, TimestampNTZ, and Time.
func (Value) ObjectValue ¶
ObjectValue returns the Object. Panics if not an object.
func (Value) Type ¶
func (v Value) Type() PrimitiveType
Type returns the PrimitiveType of the value. For non-primitive values (objects, arrays, short strings), it returns -1.
func (Value) Write ¶
func (v Value) Write(w ValueBuilder)
Write replays the value as a sequence of events on w. It is the bridge from the tree representation to any ValueBuilder.
type ValueBuilder ¶
type ValueBuilder interface {
Null()
Bool(v bool)
Int8(v int8)
Int16(v int16)
Int32(v int32)
Int64(v int64)
Float(v float32)
Double(v float64)
String(v string)
Binary(v []byte)
Date(days int32)
Time(micros int64)
Timestamp(micros int64)
TimestampNTZ(micros int64)
TimestampNanos(nanos int64)
TimestampNTZNanos(nanos int64)
UUID(v uuid.UUID)
Decimal4(unscaled int32, scale byte)
Decimal8(unscaled int64, scale byte)
Decimal16(unscaled [16]byte, scale byte)
// BeginObject starts an object value. Calls until the matching EndObject
// must alternate Field and value events.
BeginObject()
// Field names the next value written as a field of the innermost open
// object.
Field(name string)
// EndObject closes the innermost open object.
EndObject()
// BeginArray starts an array value. Every value written until the
// matching EndArray is one element.
BeginArray()
// EndArray closes the innermost open array.
EndArray()
// Err returns the first error encountered, or nil.
Err() error
}
ValueBuilder is a streaming event sink for variant values. Callers describe a value as a sequence of events — scalars, or containers bracketed by Begin/End calls — and implementations consume the events without ever materializing the value as a tree.
Exactly one value must be written: a single scalar event, or a single balanced container. Inside an object, every value must be preceded by a Field call naming it, and field names must be unique within their object (duplicates are an error). Errors (both misuse of the event sequence and implementation-specific failures) are sticky and reported by Err; events after an error are ignored.
ValueBuilder is implemented by Builder, which encodes the events to variant binary, and by parquet.VariantColumnWriter, which shreds the events directly into parquet column buffers.
Source Files