Part 6: Governance — Evals, Policy, Deployment Strategy (The missing layer)
In Part 5 we instrumented the fleet. We covered distributed tracing for agent reasoning chains, structured logs for tool calls and MCP invocations, cost attribution per agent run, and OTEL integration for unified observability across frameworks. Now we have visibility — but visibility without control is just anxiety. Part 6 adds the control layer.
1. Why Governance is the Missing Layer
Every team building with AI agents eventually hits the same wall. You have frameworks like LangGraph, CrewAI, or OpenAI Agents SDK to define how an agent reasons. You have harnesses like Harness Agent DLC, Microsoft Agent 365, or Amazon Bedrock AgentCore to define where and how reliably it runs. Yet neither answers the most critical enterprise question: what data is this agent allowed to read, what actions can it take, and who approved this version for production?
This is exactly the gap called out in Atlan's recent analysis of the data governance blind spot. Frameworks govern reasoning. Harnesses govern execution. Neither, by default, governs data access. An agent that can call a tool to read your entire Snowflake warehouse or exfiltrate via an unapproved MCP server isn't a productivity boost — it's a breach waiting to happen.
Gartner now forecasts that 40% of enterprise AI agent projects will be abandoned or rebuilt by 2027 due to inadequate governance. Not because the models failed, but because security, legal, and platform teams couldn't answer basic questions: Which version of this prompt is live? What evals did it pass? What policies blocked it last week?
We are shifting from building an agent to governing fleets of agents. A single customer support agent might be manageable with code review. A fleet of 50 specialized agents — research, coding, finance reconciliation, IT ops — each with different tools, data access, and risk profiles, is an operational problem that looks a lot like microservices in 2016.
For agents, governance means three things: quality is measurable (evals), policy is enforceable as code (OPA, Apigee), and deployment is progressive and reversible (canary, blue-green, manifest-driven). Without these, you don't have a platform. You have a collection of scripts with API keys.
2. Evals - Making Agent Quality Measurable
Traditional software has CI: tests pass or fail. Agents have evals: quality is probabilistic. Change the underlying model from Claude 3.5 to 4.1, keep the same prompt, and your agent might get 12% better at tool use but 8% worse at refusing out-of-scope requests. Without evals, you won't know until a customer does.
This is why Harness AI Evals, launched July 21, 2026 as part of Harness Agent DLC, matters. It reframes quality as a first-class platform feature. You define eval datasets (golden traces, edge cases, adversarial prompts), wire up scoring functions (LLM-as-judge, regex, exact match, custom Python), and set quality gates that automatically catch regressions whenever the agent or model changes.
The workflow looks like this: Your agent PR updates a system prompt. The Harness pipeline triggers the eval harness. It runs against your curated 500-case dataset. Scoring functions grade tool-call accuracy, final answer correctness, policy compliance, and cost. If the new prompt drops below 94% on the "no PII leak" eval, the gate fails. Merge blocked. No human needed to spot it.
This is fundamentally different from traditional pass/fail CI. An eval score of 0.87 on a reasoning benchmark isn't failure — it's signal. You track it over time. You set thresholds. You compare canary vs baseline. You need eval harnesses that match the agent's job: SWE-bench for coding agents, GAIA for general assistant reasoning, BFCL (Berkeley Function Calling Leaderboard) for tool-use accuracy. Harness Agent DLC lets you plug these in as pipeline steps, not one-off notebooks.
The teams that win at enterprise scale aren't those with the best prompts. They're those who can answer, in one dashboard, "Did last night's model update regress our finance agent?"
3. Policy as Code - OPA Guardrails and API Protection
Evals tell you if an agent is good. Policy as code tells you if it's allowed to run.
The pattern is borrowed from Kubernetes: OPA (Open Policy Agent) guardrails that evaluate every deployment against declarative policy. "No agent may have access to both prod DB and external internet in the same run." "All finance agents must have PII redaction enabled." "Agents cannot use MCP servers not on the allowlist." These are Rego policies, versioned in Git, enforced at deploy time.
Harness pioneered this for software delivery, and now applies the same primitives to agents. Harness pipelines can apply canary releases, manual approvals, and OPA guardrails to managed agent runtimes like Amazon Bedrock AgentCore and Google Agent Runtime. Your agent isn't just a Lambda you push — it's an artifact that goes through stages: dev → eval gate → OPA policy check → canary 5% → approval → 100%. The same pipeline that protects your Kubernetes deployments protects your agent fleet.
The second piece is API and tool governance. On July 30, 2026, Harness announced integration with Apigee API Management for discovering and protecting agents, AI services, tools, and MCP workflows. This is critical because agents don't just call APIs — they are APIs, and they expose tools that are themselves APIs. Apigee gives you discovery (what agents exist?), protection (rate limiting, auth, threat detection), and audit for every tool call an agent makes.
Add the AppSec Alliance announcement from July 22, 2026, and the picture completes: Harness Agent DLC, combined with Apigee and Harness STO (Security Testing Orchestration), can scan agent tool definitions for secrets, enforce mTLS for MCP connections, and block deployment if an agent declares a tool that bypasses DLP.
Without policy as code, your MCP governance is a spreadsheet. With it, it's an automatic gate.
4. Deployment Strategies for Agents
Deploying an agent is not like deploying a REST API. An API change breaks deterministically. An agent change degrades probabilistically. That demands progressive delivery.
Canary releases for agents are now baseline. You deploy v2 of your support agent to 5% of traffic. You compare eval scores, user satisfaction, tool error rate, and cost vs v1 (still serving 95%). If v2 hallucinates more on refund policy, you roll back — not because of a 500 error, but because of a 4% eval regression. Harness Agent DLC supports this natively with automated canary analysis.
Blue-green is useful for high-stakes agents like finance reconciliation. Two identical environments, only one live. You run full eval suite and manual approval on green, then flip the router. Progressive rollouts extend canary: 5% → 25% → 50% → 100% with eval gates at each step.
The harder problem is versioning. An agent has three mutable layers: prompt (system + templates), harness (orchestration logic, retry, memory, DLC runtime config), and model (Claude, Gemini, GPT-5). Changing any one can regress quality. Best practice is a manifest that pins all three plus tool versions.
Harness uses a manifest + RunConfig pattern: a workspace description that declares agent name, prompt artifact (versioned in Git or Harness Artifact Registry), harness version, model ID, allowed MCP servers, and eval datasets. RunConfig defines deployment strategy, policy sets, and rollback criteria. Change one line, trigger evals, enforce policy, deploy progressively.
This is how enterprises scale. KPMG and Microsoft announced June 9, 2026 that they scale trusted enterprise AI agents via Agent 365 and Copilot using exactly this model — governed agent workspaces with standardized deployment manifests, eval gates, and policy enforcement. An agent is no longer a notebook; it's a versioned, governed release artifact.
5. The Agentic AI Foundation and Standardization
Governance doesn't scale if every vendor invents its own protocol. The industry move to standardize around MCP (Model Context Protocol) is a governance win.
In December 2025, Anthropic donated MCP to the Agentic AI Foundation (AAIF), co-founded with Block and OpenAI, and now joined by Google, Microsoft, AWS, Cloudflare, and Bloomberg as founding members. The goal: keep the agent-tool protocol open, versioned, and secure. AAIF now stewards MCP spec, A2A (Agent-to-Agent) protocol drafts, and security profiles for MCP servers.
Nvidia's alliance announcement added momentum: open models, open weights, open datasets, and open research on agent harness evaluation. When the harness research itself is open, governance tooling can be shared rather than siloed per platform.
Even Meta Harness — rumored to be demoed privately at YC Startup School 2026 — is reportedly built on AAIF standards, with native OPA and eval gates. The message is clear: the next wave of agent platforms will compete on governance, not just reasoning.
6. Production Governance Checklist
If you manage more than three agents, you need a checklist. Not a doc — an automated pipeline. Here's the five-step governance loop every fleet should implement:
- 1) Define Evals: Create a dataset per agent capability. 100+ cases for core flows, 20+ adversarial cases for policy. Wire scoring functions (LLM-as-judge + deterministic). Set threshold: e.g., ≥92% correctness, ≥98% policy compliance. Store in Harness AI Evals or your eval store.
- 2) Set Policy as Code: Write OPA Rego policies for data access, allowed MCP servers, required DLP steps, cost limits per run, and human-in-the-loop requirements. Enforce via Harness Pipeline + Apigee. Block on violation.
- 3) Enforce Deployment Gates: Manifest + RunConfig versioning for prompt/harness/model. Canary 5% with auto-rollback on eval regression or error spike. Blue-green for financial/legal agents. Approval step for prod promotion.
- 4) Audit MCP Tool Calls: Log every tool invocation with agent ID, user context, data accessed, and policy decision. Ship to SIEM via OTEL. Use Apigee to discover shadow agents and unapproved tools. Rotate secrets via Harness Secrets Manager.
- 5) Monitor Fleet: Fleet-level dashboard: eval drift over time, policy block rate, cost per successful task, tool failure rate. Alert on eval regression >3% week-over-week or new MCP server appearance.
| Capability | Harness DLC | Microsoft Agent 365 | Google Agent Runtime | Bedrock AgentCore |
|---|---|---|---|---|
| Evals | Native AI Evals (July 2026) + BFCL/GAIA/SWE-bench pipeline steps, quality gates | Eval harness via Foundry, Copilot metrics | Vertex AI Evals, auto-rater | Bedrock Evals, model-only scoring |
| Policy | OPA guardrails in pipeline + Apigee integration (July 30 2026), STO | Entra + Purview policies | Apigee + Model Armor policies | Guardrails for Bedrock + IAM |
| Deployment | Canary, blue-green, progressive with manifest/RunConfig, auto-rollback on eval | Agent 365 rollout via Intune/Copilot | Progressive via Vertex Pipelines | Canary via CodeDeploy for agents |
| MCP Governance | Apigee discovery + allowlist enforcement, tool audit | MCP registry in Foundry, Entra auth | MCP Toolbox + Apigee governance | MCP + Action Groups governance |
| Audit | OTEL traces + full tool-call audit + cost ledger | Purview + Copilot audit logs | Cloud Audit Logs + OTEL | CloudTrail + Bedrock logs |
Takeaway — Governance is the Platform
Observability showed you what your agents did. Governance decides what they are allowed to do, and proves it.
- Evals turn quality from vibes to metrics — track regression per prompt, harness, and model change.
- Policy as code (OPA + Apigee) moves security left: block unapproved tools before they reach prod, not after an incident.
- Deployment strategies for agents must be progressive and eval-aware; direct push is tech debt at fleet scale.
- AAIF and MCP standardization means governance tooling can finally be portable across Harness DLC, Agent 365, Google, and Bedrock.
FAQ
What's the difference between traditional CI tests and AI evals?
CI tests are deterministic pass/fail — your function returns 42 or it doesn't. Evals are probabilistic scoring — your agent's answer is graded on a spectrum (correctness 0-1, policy compliance, cost). You set thresholds and track drift over time rather than expecting 100% pass. Harness AI Evals automates this as quality gates in your pipeline.
How do OPA policies work for agents?
OPA policies are Rego rules evaluated at deployment. For agents, they check: is this MCP server on the allowlist? Does this agent have PII redaction enabled? Is cost per run < $0.50? If any policy denies, the Harness pipeline blocks promotion to prod. Same pattern as Kubernetes admission controllers.
Should I version prompts, harness code, and models separately?
Yes, but pin them together in a manifest. Version prompt v1.4.2, harness DLC v0.9.1, model claude-3-5-sonnet-20241022 in a single RunConfig. This gives you reproducible evals and rollback — you can say "revert to manifest 2026-07-20" and restore all three layers atomically.
Do I need Apigee if I'm already using OPA?
They solve different layers. OPA blocks bad deployments. Apigee discovers and protects running agents and tools — rate limiting, auth, threat detection for every MCP tool call. For fleet governance, you want both: OPA at deploy time, Apigee at runtime.
Next up in Part 7: Choosing Your Harness — we compare DLC, Agent 365, Agent Runtime, and Bedrock AgentCore decision criteria for your stack.
[Part 6 Complete. Say "Go" or "Proceed" to generate Part 7.]
No comments:
Post a Comment