The directory of AI wrapper tools and LLM middleware
Almost no production application talks to a large language model directly. Between your code and the raw completion endpoint sits a growing stack of wrapper tools — the libraries, gateways, and platforms that make LLMs reliable, observable, affordable, and safe enough to ship. WrapDirectory catalogs that middleware layer in one place, so engineering and platform teams can compare options without wading through a dozen marketing sites. Every listing here is a real, verifiable project or product with a link to its official source.
We organize the landscape into five practical categories. LLM frameworks — such as LangChain, LlamaIndex, DSPy, Haystack, Semantic Kernel, CrewAI, and Pydantic AI — give you the building blocks for chains, retrieval-augmented generation (RAG), and multi-agent orchestration. Routing and gateways — LiteLLM, OpenRouter, Portkey, Kong AI Gateway, vLLM, and Ollama — sit in front of one or many model providers to normalize APIs, balance load, cache responses, control spend, and remove vendor lock-in. Observability tools — Langfuse, LangSmith, Helicone, Phoenix, Weights & Biases, Braintrust, AgentOps, OpenLLMetry, and Promptfoo — capture traces, run evaluations, and track cost and latency so non-deterministic behavior becomes debuggable. Prompt management platforms like PromptLayer and Agenta version prompts and decouple them from application code. And guardrails projects such as Guardrails AI and NVIDIA NeMo Guardrails enforce safety, structure, and topical boundaries on what models say and do.
Why does this middleware matter? A bare model call is a single point of failure: if your provider has an outage, your product goes down; if a prompt drifts, quality silently degrades; if an agent loops, your bill explodes. The tools in this directory address exactly those risks. A routing proxy adds automatic fallbacks and retries across providers. An observability platform turns an opaque chain of model calls into an inspectable trace with token counts and evaluation scores. A guardrails layer blocks prompt injection and PII leakage before it reaches a user. A structured-output library like Instructor, Outlines, or Mirascope guarantees the model returns valid JSON the first time instead of breaking your parser. Adopted together, these wrappers are what separate a weekend demo from a system that survives real traffic.
The market moves quickly, with new open-source projects and commercial platforms appearing every month and existing ones overlapping into adjacent categories — many gateways now bundle observability and guardrails, and many frameworks ship their own tracing. WrapDirectory tracks these shifts and keeps each entry current with a plain-language editorial summary of what the tool does, who builds it, and where it fits in the stack. Use the category filters and search above to narrow the field to your use case, whether you are choosing your first framework, hardening an agent for production, or standardizing tooling across an engineering organization. Note too that the categories are guidance rather than hard walls: a project such as Portkey spans routing, caching, observability, and guardrails at once, while LangChain, LlamaIndex, and Langfuse each blur the line between framework and platform — so it is worth reading the editorial summary for any tool you shortlist rather than relying on its label alone. Many of the strongest production stacks combine several of these tools, pairing a framework for orchestration with a gateway for resilience, an observability platform for evaluation, and a guardrails layer for safety. Browse the full listing below and follow the official links to dig deeper.
Llm Frameworks
Stanford framework for programming LLMs — automated prompt optimization replaces hand-crafted prompt engineering.
DSPy is a framework from the Stanford NLP group for programming language models rather than hand-crafting brittle prompts. Developers declare the input and output behavior they want using typed signatures and composable modules, and DSPy's optimizers (such as MIPROv2 and BootstrapFewShot) automatically compile and tune the underlying prompts and few-shot examples against a metric. This shifts prompt engineering from manual trial-and-error to a systematic, measurable optimization process, and the project has become a leading approach for building self-improving LLM pipelines.
Llm Frameworks
python stanford prompt-optimization open-source
Microsoft's guidance-ai library — constrain and control LLM generation with a programmatic templating language.
Guidance is an open-source library, originally created at Microsoft and now stewarded by the guidance-ai organization, that gives developers fine-grained control over text generation from large language models. Instead of issuing a single prompt and parsing the result, Guidance interleaves generation with control structures — loops, conditionals, regex constraints, and selections — so the model is steered to produce exactly the structure required. This constrained-decoding approach reduces wasted tokens, improves reliability for structured output, and works across both local and hosted models.
Llm Frameworks
python constrained-generation microsoft open-source
Python library for structured LLM outputs using Pydantic — turn LLM responses into type-safe objects reliably.
Instructor, created by Jason Liu, is a popular Python library that makes structured extraction from large language models simple and reliable by building on Pydantic. Developers define the desired schema as a Pydantic model, and Instructor patches the LLM client so responses are validated, coerced into typed objects, and automatically re-asked when validation fails. It works on top of function calling and JSON modes across providers including OpenAI, Anthropic, and Gemini, and has been ported to TypeScript, Go, Ruby, and other languages, making it a de facto standard for data extraction.
Llm Frameworks
python structured-output pydantic open-source
The most widely used open-source framework for building LLM applications — chains, agents, RAG, and integrations.
LangChain is the most widely adopted open-source framework for building applications on top of large language models, with libraries in both Python and JavaScript/TypeScript. It pioneered the abstraction of "chains" that compose prompts, models, memory, and tools into reusable pipelines, and it ships hundreds of integrations spanning vector stores, document loaders, and model providers. Maintained by LangChain, Inc. (which also builds LangSmith and LangGraph), it remains a default starting point for RAG systems and agentic workflows.
Llm Frameworks
python framework open-source rag
Data framework for LLM apps — connect, structure, and query private data with agentic RAG workflows.
LlamaIndex is a data framework purpose-built for connecting large language models to private and domain-specific data. It provides ingestion connectors, indexing structures, and query engines that make retrieval-augmented generation reliable at scale, and it has expanded into agentic workflows through LlamaAgents and the managed LlamaCloud parsing service. Created by Jerry Liu and maintained by the LlamaIndex team, it is available in both Python and TypeScript and is one of the most popular RAG-focused libraries in production use today.
Llm Frameworks
python rag data-indexing open-source
Structured text generation library — constrain LLM outputs to specific formats using regex and JSON schemas.
Outlines, developed by the team at .txt (dottxt), is a Python library for guaranteed structured generation from language models. It compiles regular expressions, JSON Schemas, Pydantic models, and context-free grammars into finite-state machines that mask the model's token logits during decoding, so the output is forced to conform to the requested format without retries or parsing failures. Because the constraints operate at the sampling level, Outlines adds negligible overhead and works with local open-weight models served through vLLM, transformers, and similar backends.
Llm Frameworks
python structured-generation regex open-source
Routing Proxy
Unified API layer for 100+ LLMs — call any model with OpenAI-compatible syntax; load balancing and fallbacks.
LiteLLM, built by BerriAI, is an open-source unified interface that lets developers call more than 100 large language model providers using the familiar OpenAI completion syntax. It ships both as a lightweight Python SDK and as a self-hostable proxy server (gateway) that adds load balancing, automatic fallbacks, retry logic, spend tracking, and virtual API keys for teams. By normalizing inputs and outputs across providers such as OpenAI, Anthropic, Azure, Bedrock, and Vertex AI, it removes vendor lock-in and is widely used as the routing layer in production LLM stacks.
Routing Proxy
proxy multi-model openai-compatible open-source
Unified API for accessing 200+ AI models from a single endpoint — route to best model by price, speed, or quality.
OpenRouter is a hosted inference marketplace that exposes hundreds of models from dozens of providers behind a single OpenAI-compatible API endpoint. Developers send one request and OpenRouter handles authentication, billing, and routing — automatically falling back to alternate providers when one is down and letting users prioritize by price, latency, or throughput. It aggregates models from OpenAI, Anthropic, Google, Meta, Mistral, and many open-weight hosts, provides unified usage analytics, and removes the need to hold separate accounts and keys with each model vendor.
Routing Proxy
proxy multi-model api routing
AI gateway with routing, caching, observability, and guardrails — drop-in middleware for LLM API calls.
Portkey is an AI gateway and control panel that sits between applications and language model providers. Its open-source gateway routes traffic across 250-plus models with load balancing, conditional routing, semantic caching, automatic retries, and fallbacks, while the observability suite logs every request with cost, latency, and tracing. Portkey also layers in guardrails, prompt management, and budget controls, giving platform teams a single place to enforce reliability and governance policies. It is available as a hosted service and as a self-hostable, enterprise-deployable component.
Routing Proxy
proxy multi-model observability caching
Observability
Enterprise AI evaluation and observability — run evals, trace LLM calls, and monitor production AI quality.
Braintrust is an end-to-end platform for evaluating and shipping AI products, used by engineering teams to bring rigor to non-deterministic LLM behavior. It centers on evals — scoring model outputs against datasets with code-based or LLM-as-a-judge scorers — alongside logging and tracing of production calls so regressions are caught before release. Its Playground lets engineers and domain experts compare prompts and models side by side, and the Loop assistant helps author scorers. Braintrust is offered as a hosted product with options for on-premise and hybrid deployment.
Observability
eval observability tracing enterprise
LLM observability platform — one-line integration for logging, caching, rate limits, and cost tracking.
Helicone is an open-source LLM observability platform that integrates through a single line of configuration by routing requests through its proxy or async logging SDK. Once connected, it captures every request and response with full token counts, latency, and cost, and adds production features such as response caching, rate limiting, retries, and user-level analytics. Helicone's gateway model makes it provider-agnostic, and teams use it to monitor spend, debug prompts, and segment usage by customer. It is available both as a hosted service and a self-hosted deployment.
Observability
observability logging caching open-source
Open-source LLM observability platform from Arize — traces, evals, and prompt playground in one tool.
Phoenix is the open-source LLM observability and evaluation platform from Arize AI. It captures traces of LLM, retrieval, and agent calls using the OpenInference standard built on OpenTelemetry, then surfaces them in an interactive UI for debugging RAG pipelines, spotting hallucinations, and analyzing token usage. Phoenix bundles a library of LLM-as-a-judge evaluators and a prompt playground so teams can iterate without leaving the tool, and it can run locally in a notebook or be self-hosted, complementing Arize's enterprise monitoring product.
Observability
observability tracing evals open-source
ML experiment tracking and LLM observability — W&B Weave for LLM trace logging, evals, and dataset management.
Observability
mlops experiment-tracking eval fine-tuning
Guardrails
Open-source framework for adding safety guardrails to LLM outputs — validate, structure, and correct AI responses.
Guardrails
guardrails validation python open-source
Prompt Management
Prompt management and LLM request logging — version prompts, track costs, and replay LLM requests.
Prompt Management
prompt-management logging versioning collaboration