LiteRT October Release

The following are the release notes for the LiteRT October release.

LiteRT Release Notes: Version 2.3.0

Summary

The LiteRT 2.3.0 release represents a leap in on-device AI efficiency, particularly for Large Language Models (LLMs) and cross-platform GPU/NPU utilization.

API and Platform updates

  • Swift Package for iOS & macOS - New type-safe Swift APIs to run LiteRT models with hardware acceleration (developer guide).
  • Support Library - Migrated from TensorFlow Lite Support to LiteRT Support (migration guide).
  • Android - minSdk raised to 24; GPU (litert-gpu) and vendor-specific NPU libraries now ship as separate Maven packages (NPU sample).

Hardware acceleration

Performance and Memory optimizations

  • FlashAttention on Metal - Up to 38% faster decode, up to 2.98x faster prefill, and >4 GB less peak RAM at 32K context.
  • Fused LLM kernels - New QKV-RMSNorm-RoPE and SwiGLU GPU kernels for ~12% faster decode.
  • GPU memory - Metal memory residency on iOS/macOS avoids swap paging; zero-copy mmap for the program cache lowers peak RAM.

Validation / Testing Infrastructure

  • ATS - Coverage expanded to all core single ops and to macOS (Metal and WebGPU).

LiteRT-LM Release Notes: v0.18.0

  • New Model Release (EmbeddingGemma 2): Shipped multimodal EmbeddingGemma 2 supporting text, vision, and audio embeddings with Matryoshka dimension truncation across Python, Kotlin, Swift, Web (JavaScript), and C++. Check out our DevSite documentation for more details!
  • CLI & Developer Experience: Added fast model imports (litert-lm import) and an OpenAI-compatible /v1/embeddings endpoint (litert-lm serve) for multimodal embedding models.
  • Model Info API: Added ModelInfo and litert-lm describe for full model introspection—covering metadata, capabilities, and runtime requirements before loading.
  • NPU & GPU Acceleration: Enabled dynamic on-demand KV cache growth on NPU and attention mask pruning optimizations on GPU.

LiteRT-LM Release Notes: v0.18.1

  • Fixed an ABI-incompatibility issue that prevented running EmbeddingGemma 2 when building LiteRT-LM from source.

LiteRT Quantizer Release Notes: v0.10.0

  • Old Revision: v0.9.0 (Commit a4ae302)
  • New Revision: v0.10.0 (Commit 7e6af26)
  • Repository: google-ai-edge/ai-edge-quantizer

AI Edge Quantizer will be rebranded to LiteRT Quantizer and become a part of LiteRT GitHub repo in the next release.

Quantization Algorithms & Recipes

  • New experimental OSCAR algorithm (AlgorithmName.OSCAR) for FULLY_CONNECTED. It combines activation-aware per-channel activation scaling with optimal weight clipping.
  • Simpler recipe validation. weight_tensor_config is no longer required for ops without weights (e.g. model INPUT/OUTPUT), so you don't need a dummy weight config anymore.

Tooling & Model Formats

  • uint8 model inputs/outputs. The new TensorDataType.UINT lets recipes set uint8 for model INPUT/OUTPUT tensors, which some hardware accelerators require.

Documentation

  • README rewrite. It now covers key features, the CLI, the end-to-end workflow, recommended recipes, and calibration modes/APIs.
  • Corrected operator support table, now split into ops with weights and activation-only ops, removed unsupported configs.

LiteRT Torch Release Notes: v0.10.0

🔹 Vision & Generative Pipelines

  • Added Gemma 4 Audio Encoder export and .litertlm packaging support (--task=multimodal_lm), bundling vision and text components.
  • Added support for heterogeneous per-layer KV cache dimensions (per_layer_config with differing head_dim and num_key_value_heads across sliding and full attention layers, such as in Gemma 4 12B).
  • Speech & Audio Models (Qwen3-TTS & Qwen3-ASR):
    • Added Qwen3-TTS text-to-speech model export support (--task=text_to_speech) in export_hf.
    • Added .litertlm bundle packaging support for Qwen3-ASR automatic speech recognition models (--task=automatic_speech_recognition --bundle_litert_lm=True).
  • Added automatic pad_token metadata population from the tokenizer, plus model-type mappings and MiniJinja chat template normalization

🔹 NPU & Acceleration Passes

  • Export_hf Added --cache_lengths flag to export multiple static KV cache length signatures into a single model, and updated per-layer embedder export to generate signatures for all configured prefill_lengths.

🔹 Converter Legalizations & Dialects

  • 4D Attention View Rank Folding
    • Added an algebraic FX pass that folds unsqueeze -> permute -> squeeze view chains (such as in torch.nn.MultiheadAttention QKV unpacking) into a single permute, keeping intermediate tensors within the rank-4 limit of the ML Drift GPU delegate.
  • Added support for nn.Conv2d with padding_mode='reflect' and padding_mode='replicate', lowering aten.reflection_pad2d and aten.replication_pad2d to TFLite MIRROR_PAD composites across both NCHW and NHWC layout-optimized paths instead of GATHER_ND subgraphs.
  • Promoted rank-3 1D convolutions to rank-4 2D convolutions in pre-conversion FX decompositions so NHWC layout optimization preserves CONV_2D bias fusion
  • Lowered contiguous static aten.as_strided views to ravel -> slice -> reshape instead of GATHER_ND, and folded batch dimensions of rank >= 3 non-contiguous aten.linear inputs into 2D matmuls (addmm/mm) to avoid rank-5/6 BROADCAST_TO + BATCH_MATMUL subgraphs.

🔹 Core Bug Fixes

  • Fixed runtime_fold() to retain the folded MLIR module, preserve op order when pairing folded constant arrays to prevent silent buffer mispairing, and fold dequantized constant weight expressions (fold_dequantize=True).
  • Fixed _promote_types_for_binary_op in torch_tfl decompositions to use torch.result_type, preventing Python float scalars from being truncated to integer tensor dtypes (e.g., int_tensor * 0.5 becoming int_tensor * 0).