Opik: Technical Architecture for Open-Source LLM Observability

Core Takeaways

Opik is an open-source LLM observability, evaluation, and monitoring platform developed by Comet. It addresses data privacy and overhead challenges common in closed-source observability platforms by allowing developers to self-host the complete backend stack—including trace collectors, evaluation engines, prompt management, and web dashboards—under the Apache-2.0 license.

The primary technical tradeoff when adopting Opik is operational responsibility: self-hosting requires managing storage and service orchestration via Docker Compose or Kubernetes (Helm) in exchange for absolute data sovereignty, low telemetry latency, and zero per-trace vendor fees.

Agent Tracing and Telemetry Pipeline

In multi-step agentic workflows, capturing nested tool calls, retrieval steps, and model completions requires lightweight latency overhead. Opik achieves trace capture through Python and TypeScript SDKs, as well as native OpenTelemetry support for cross-language environments like Java, Ruby, and .NET, as detailed in the Opik README.

Tracing a function in Python requires wrapping the execution context with the @track decorator:

from opik import track

@track
def run_agent_step(prompt: str) -> str:
    # Executes LLM call or tool integration
    return f"Processed: {prompt}"

When @track wraps a function, the SDK constructs nested trace trees across function boundaries. Tracing spans are buffered asynchronously in client memory before transmission to the backend, mitigating inline performance bottlenecks during LLM interactions. For systems leveraging frameworks like LangChain, LlamaIndex, Autogen, Google ADK, CrewAI, or Flowise AI, Opik provides automated callbacks that extract prompt templates, token metadata, and span hierarchies.

Additionally, Opik supports developer-assisted debugging via the Model Context Protocol (MCP). Running uvx opik mcp configure enables AI coding tools such as Claude Code, Cursor, and VS Code Copilot to query traces, run evaluations, and analyze model behavior directly within the development environment.

Automated LLM Evaluation and Guardrails

Evaluation in Opik operates both offline (during development and CI/CD) and online (continuously on production streams).

1. Offline Datasets and Experiments: Teams create test datasets and execute candidate models or prompt variations against pre-defined metrics. Opik integrates directly with PyTest, enabling teams to enforce evaluation assertions on every commit.
2. LLM-as-a-Judge Metrics: Opik provides pre-built evaluators for critical quality dimensions:
   • Hallucination Detection: Assesses whether model responses contain ungrounded claims relative to reference context.
   • RAG Assessment: Evaluates Answer Relevance and Context Precision to tune vector retrieval pipelines.
   • Moderation: Flags harmful or policy-violating outputs.
3. Production Online Evaluation & Guardrails: In production environments, online evaluation rules run asynchronously on incoming traces to log quality metrics over time without delaying model responses. The optional Opik Guardrails component can intercept unsafe prompt inputs or model outputs prior to client response.

Deployment Architecture and Infrastructure Modes

Opik provides flexible deployment targets ranging from local container clusters to production enterprise Kubernetes instances:

• Local Development: Managed using the ./opik.sh utility script. The script provides modular options to start specific sub-components:
  • ./opik.sh boots the full platform suite.
  • ./opik.sh --infra starts foundational data stores and caching layers.
  • ./opik.sh --backend launches backend collection and evaluation services.
  • ./opik.sh --guardrails activates guardrail enforcement containers.
  • ./opik.sh --verify performs health checks on running containers.
• Kubernetes Production: Production self-hosting uses the official Opik Helm chart on Kubernetes, deploying microservices configured to run under non-root permissions for container security compliance.
• Managed Cloud: Comet provides a fully managed SaaS instance on Comet.com for teams seeking zero-maintenance infrastructure.

Comparison with Alternative Observability Platforms

As outlined in the Opik repository metadata, Opik distinguishes itself across licensing and hosting paradigms:

• LangSmith & Arize AX: Closed-source self-hosting that requires enterprise commercial contracts.
• Phoenix & W&B Weave: Source-available or enterprise-restricted self-managed backends.
• Langfuse: MIT-licensed core platform, though certain enterprise modules are closed.
• Opik: Entire stack—backend server, evaluation engine, web app, prompt playground, agent optimizer—is 100% Apache-2.0 licensed and freely self-hostable.

Sources

• Primary Documentation:
  • Opik GitHub Repository
  • Opik Official README