LLM Client Libraries and SDKs

Cua supplies desktops and automation tools for developers building AI agents that operate native apps and browsers. You bring your own agent and model; Cua provides the computer environment and ways to interact with it.

LiteLLM is an AI gateway for platform teams managing model access across applications and providers. It puts cloud and self-hosted models behind a shared interface so developers can switch providers without rewriting their integration.

LiteRT-LM is Google's open-source framework for developers building apps that run language models on the user's device. It uses LiteRT for local inference across mobile devices, desktops and browsers, with GPU and NPU acceleration. The project uses the Apache 2.0 license.

PyOD is a Python library for detecting unusual records and behavior in data on your own machine or server. A shared fit/predict interface lets developers and data scientists compare classical and deep learning anomaly detectors without changing the surrounding application for each algorithm. Its code uses the BSD 2-Clause license.

LM Format Enforcer is an open-source Python library for developers who need a local LLM to return structured data their applications can parse. It constrains text during generation, so a prompt's formatting request doesn't have to carry the whole burden. The library uses the MIT license.

Bifrost is an AI gateway for developers and teams building applications that use local LLMs or multiple cloud providers. It gives those applications a shared API for model access and a central place to manage usage.

chromem-go is a vector database for Go developers adding semantic search or retrieval augmented generation (RAG) to an application. It runs inside the app, so it doesn't need a separate database server.

LLM Compressor prepares models for deployment with vLLM by reducing their storage and memory requirements through quantization. It's for developers working with local checkpoints or models from Hugging Face repositories.

Haystack is a Python framework for developers building AI agents, document search and applications that answer questions using their own data. Its pipelines give teams control over the information a model receives and the steps that produce an answer.

npcpy is a Python library for developers and researchers building AI agents and applications with language models. It supports local LLM backends as well as cloud providers, so projects can choose where model requests run.

Quivr is a content search and monitoring engine for developers building applications around incoming material and topics they want to follow over time. It's at an evaluation stage, and the project advises against production use because its API may change without notice.

ExtractThinker is a Python library for developers who want to extract structured data from documents using models running locally through Ollama. It also supports OpenAI, Anthropic and Groq. With a cloud provider, extraction sends document content to that service, requires provider credentials and may incur charges.

Safetensors is a file format and library for developers storing and loading AI model weights on their own hardware or servers. It addresses the arbitrary code execution risk of pickle-based PyTorch files.

LangChain is an open-source framework for developers building AI agents and LLM applications, including apps that run models locally through Ollama. Its Python ChatOllama integration connects to open-weight models such as gpt-oss. You write the application in code, while Ollama handles local inference.

Composio's local CLI connects coding agents such as Claude Code to a hosted service for app actions and authentication. It's for people adding integrations to an assistant or building their own agents. The CLI runs on your machine, but requires a Composio account and internet access; model processing depends on the connected agent and its provider.

LazyLLM is a Python framework for developers building applications with multiple AI agents. It brings application workflows, model training and evaluation together so teams can refine a prototype using task-specific data.

Rivet AgentOS is a runtime for developers building applications where AI agents need a working environment for code and tools. It runs inside an existing backend process, with WebAssembly isolation instead of a separate virtual machine for each agent.

Nexa SDK is an on-device AI framework for developers building apps that run models on users' hardware. It covers language, vision and audio tasks through a shared interface for local inference.

PaddleNLP is a Python toolkit for developers training and deploying language models on their own infrastructure. Built on PaddlePaddle, it covers pretraining, fine-tuning, model compression and inference across several accelerator families.

Ollama Python is a client library for developers adding model conversations to Python applications. It connects to a running Ollama service for local use or to Ollama's cloud service for larger models.

smol developer is a Python coding agent with a self-hosted API for generating codebases from product specifications. It's for developers prototyping applications or adding code generation to their own software. Its default generation workflow uses GPT-4 through the OpenAI Python SDK, whose standard API endpoint sends model requests to OpenAI’s cloud. Self-hosting controls the agent server, while this standard model connection remains remote.

OpenAI Agents SDK is a Python framework for developers building tool-using agents with local models or cloud APIs. It supports gpt-oss through Ollama, with LM Studio also documented as a local model option. OpenAI models use the remote Responses API by default and require an API key, even when the agent runtime and workspace run on your machine.

Memori's BYODB edition is a memory library for developers who want agents to retain context in their own SQLite file or self-managed database. It generates embeddings locally, but its default Advanced Augmentation sends conversation messages and summaries to Memori's cloud API for memory extraction. The documented OpenAI connection also sends model requests to OpenAI.

fx is a coding agent for developers who prefer working in a terminal and researchers who want to embed an agent in their own software. Its shell-style interface keeps tool activity alongside the conversation and preserves terminal scrollback.

Sandbox Agent lets developers control coding agents from their own applications through a shared API. It's for teams building agent-powered software that needs remote sessions and isolated code execution without a separate integration for each agent.

Mirascope is an open-source development library for building AI applications with local models or cloud providers. Its documented local backends include Ollama, vLLM and LM Studio, plus direct inference through MLX on Apple Silicon Macs. Developers can use a shared model interface while keeping inference on their own hardware or sending requests to OpenAI, Anthropic or Google.

Megatron-LM is a Python framework for researchers training large language models on NVIDIA GPU infrastructure. It provides reference training scripts for experimenting with models and studying distributed training.

Rig is a Rust library for developers building AI agents and LLM applications. It provides a shared interface across model providers and vector stores for applications that need to work with different backends.

Stagehand is an open-source browser agent SDK for developers automating websites and collecting structured data in local Chrome. Model inference is separate from browser execution: direct provider calls send model context remotely, while a custom callback can connect a self-hosted model. Local runs require Chrome.

LangChain4j is a Java library for building chatbots, assistants and AI agents that use local models through Ollama. It runs within JVM applications, while Ollama handles model inference on your hardware using a CPU or GPU. You can also connect to OpenAI or Google Vertex AI, in which case model requests go to the remote provider.

Prompty is a prompt format and developer toolkit for building AI applications. It keeps prompt text, model settings, inputs and tool definitions in a portable Markdown file that developers can share across editors and language runtimes.

PageIndex is a document retrieval engine for developers building question answering over long PDFs, including financial reports, legal documents and technical manuals. It uses an LLM to select relevant sections from a document tree without a vector database or document chunking.

EverOS is a Python library and memory runtime for developers building agents that need context across sessions and applications. It keeps conversations, files and task histories in readable Markdown that you can edit and version with Git.

CascadeFlow is a library for developers who need to control AI agent spending and behavior during a run. It works inside the application, assessing model calls, tool calls and sub-agent handoffs before deciding how execution should continue.

UQLM is a Python library for developers and researchers assessing the reliability of LLM answers. Its confidence scores estimate the likelihood of errors or hallucinations; they don't verify that an answer is factually correct.

LLMux is a self-hosted AI gateway for developers who want to use local models and cloud providers through one interface while controlling where requests go. It blocks remote inference by default and logs permitted calls that leave the machine.