Building a Control Plane for AI Agents: Architecture Guide

What Makes a Control Plane?

A control plane is the management layer that sits above operational workloads. Kubernetes is a control plane for containers — it does not run your code, it orchestrates where and how your code runs. Similarly, an AI agent control plane does not execute agent logic — it connects, observes, governs, and scales agents from any framework.

The key insight: the control plane must be framework-agnostic. It cannot assume all agents use CrewAI, or LangChain, or OpenAI. It must work with any agent that can make an HTTP call.

The Five Architectural Layers

• Gateway Layer — unified proxy for all LLM and tool requests. Authentication, rate limiting, cost tracking.
• Policy Layer — 4-level policy hierarchy (platform → org → tenant → user). Model restrictions, budgets, DLP.
• Observability Layer — immutable audit trail, real-time activity feed, cost analytics, health monitoring.
• Governance Layer — approval workflows, kill-switch, role-based access control, compliance controls.
• Integration Layer — A2A, MCP, REST, webhooks. Connect agents from any framework via any protocol.

Multi-Tenant Isolation

Multi-tenancy is the most critical architectural decision. Every organization gets complete data isolation. Every tenant (workspace) within an organization gets its own dataset, its own policies, and its own agent instances.

// 3-level hierarchy with regional isolation
//
// Platform (global)
//   └── Organization (org_xxx)
//       ├── Tenant IL (tenant_xxx) → ds_tenant_il, ds_agents_il
//       ├── Tenant EU (tenant_yyy) → ds_tenant_eu, ds_agents_eu
//       └── Tenant US (tenant_zzz) → ds_tenant_us, ds_agents_us
//
// Data NEVER crosses regional boundaries
// Cross-region visibility only via UNION ALL views (admin only)

The Policy Engine

Policies cascade through 4 layers. A platform policy sets the floor (minimum security). An organization policy can be stricter but not more permissive. A tenant policy applies within a workspace. A user-level override handles exceptions.

// Policy cascade example
// Platform: max_tokens = 8192, models = [all]
// Organization: max_tokens = 4096, models = [gpt-4o, claude-sonnet-4-20250514]
// Tenant: max_tokens = 4096, models = [claude-sonnet-4-20250514] (stricter)
// User override: max_tokens = 8192 (exception for lead engineer)
//
// Effective for lead engineer: max_tokens = 8192, models = [claude-sonnet-4-20250514]
// Effective for everyone else: max_tokens = 4096, models = [claude-sonnet-4-20250514]

Audit Trail Architecture

The audit trail uses an append-only pattern in BigQuery. Every event is a new row — never UPDATE, never DELETE. This creates an immutable record that satisfies SOC 2 and GDPR requirements. Each event includes: what happened, who did it, when, which agent, which model, how many tokens, and the full request/response.

• task_timeline — every task event (created, assigned, approved, completed, failed)
• llm_gateway_requests — every LLM call (model, tokens, cost, latency, actor)
• admin_audit_log — every admin action (policy change, key creation, role assignment)
• security_events — authentication, authorization, anomalies (365-day retention)

Gateway Design Principles

• OpenAI-compatible API — standard SDK works, no custom client libraries needed
• Circuit breaker — 5 failures → open, 30s → half-open, prevents cascade failures
• Rate limiting — per-key + per-org, sliding window in Redis, configurable by key type
• Semantic cache — two-layer (Redis exact + BigQuery cosine similarity), opt-in per org
• Kill-switch — cached 5s in Redis, propagates across all gateway instances instantly
• DLP — 9 PII patterns redacted before LLM calls (configurable per org policy)

Designing for Evolution

A control plane's real test is not day one — it is year three, after every layer has absorbed new providers, new frameworks, new regulations. The durable design choices are: speak open protocols at the edges (OpenAI, JSON-RPC, A2A), keep tenancy physical not logical, and treat policy as code with versioning and rollback. Everything else can be rewritten; these three cannot.

Frequently Asked Questions

Can I start with logical multi-tenancy and migrate later?

Technically yes, practically no. Migrating live tenants to physically separated datasets is painful, risky, and never quite complete. If you expect any compliance scope, design for physical isolation from day one.

Why BigQuery for the audit trail?

Append-only semantics, cheap long-retention storage, SQL queryability for auditors, partitioning + clustering for performance on billions of rows, and built-in regional isolation. Few other stores hit all five.

How do you keep the Gateway fast with all these checks?

Authentication, rate limiting, and kill-switch live in Redis with millisecond reads. Policy evaluation is compiled once per policy version and cached. The async paths (audit logging, analytics aggregation) happen after the response is returned to the caller.