Capability 05 / AI agents
Anyone can wire four agents together and get something impressive on a Tuesday. The hard part is the tool permissions, the evaluation harness, the tracing, the cost ceiling and the failure behaviour at 2am — and whether a second agent was ever the right answer. We build agent systems that are measured before they are shipped.
Start here
Every extra agent multiplies latency, cost and the number of ways the system can be confidently wrong. We will push back on an agent swarm where a single tool-using agent, or honestly a SQL query, does the job better.
When a problem genuinely is multi-agent — separate domains, separate permissions, separate review paths, long-running work with human checkpoints — we build it properly, with explicit state, bounded loops and a supervisor that can say no.
Architecture patterns
We pick the simplest topology the problem allows, and we write down why. These are the five we build most often.
One agent, a tight set of scoped tools, a bounded loop. The default, and the right answer far more often than the conference talks suggest. Cheapest to run, easiest to evaluate, easiest to explain to a risk committee.
A router that owns the plan, delegates to domain specialists, and holds the only view of overall state. Used when domains have genuinely different data access, tone or review requirements — a billing agent should not be able to read HR tables.
Independent subtasks run concurrently against different sources, then a reducer reconciles and cites. The pattern that makes research and document-comparison work feel fast instead of glacial.
A critic agent scores the draft against explicit criteria and sends it back, at most twice. Measurably better output on drafting and code tasks — and the first thing we cut when the latency budget is tight, because it roughly doubles cost.
Long-running processes that persist state, pause for a human approval on anything consequential, and resume hours later without losing context. How an agent gets permission to touch a system of record at all.
A large share of agent requests are better served by a well-built retrieval endpoint, a deterministic pipeline with one LLM step, or a dashboard. We would rather lose the glamorous scope and keep the client.
Frameworks
The framework matters far less than the state model, the tool boundary and the evaluation harness — all of which we would build the same way in any of them. We pick for your stack, your team’s language and your governance model, then write down the trade.
Our usual default for anything non-trivial in Python. Agents are nodes over a typed shared state, control flow is a graph you can read, and checkpointing gives you resumable runs and human-in-the-loop interrupts without inventing your own persistence.
Where the data, permissions and evaluation already live in Databricks, this removes most of the integration work: Unity Catalog functions as tools, Vector Search for retrieval, MLflow logging and tracing, Agent Evaluation with judges, and deployment to Model Serving with a review app your subject-matter experts can actually use.
Microsoft’s consolidation of the Semantic Kernel and AutoGen lineages into one SDK, with first-class .NET alongside Python and a direct path into Azure AI Foundry, Entra identity and the rest of the Microsoft governance story.
Role-and-task crews get a believable multi-agent workflow running in an afternoon, and its flows give you deterministic orchestration when you need it. Genuinely useful for shaping a problem with business stakeholders in the room.
We expose tools over the Model Context Protocol where it earns its place, so the same governed capability — a Unity Catalog function, a Vector Search index, an internal API — can be consumed by whatever framework you are using this year. The tool boundary is the asset. The orchestration library is a detail, and it will change.
Tools & governance
Tools are governed objects with owners and permissions, not Python functions holding a workspace token. Retrieval filters on the caller’s entitlements, writes go through functions that validate and log, and every call is traced with the identity that made it.
-- A tool with an owner, a signature and real permissions.
CREATE OR REPLACE FUNCTION prod.agents.open_claims(
customer_id STRING COMMENT 'Customer to look up'
)
RETURNS TABLE (claim_id STRING, status STRING, amount DECIMAL(12,2))
COMMENT 'Open claims for one customer. Read only.'
RETURN SELECT claim_id, status, amount
FROM prod.claims.claim
WHERE customer_id = open_claims.customer_id
AND status IN ('OPEN', 'IN_REVIEW');
-- The agent gets this function. Not the table.
GRANT EXECUTE ON FUNCTION prod.agents.open_claims
TO `sp-support-agent`;
-- No table grant. Nothing to escalate to.from langgraph.graph import StateGraph, START, END
class State(TypedDict):
question: str
findings: Annotated[list, add]
steps: int # bounded, always
escalate: bool
def route(state: State) -> str:
# Loops end. That is not negotiable.
if state["steps"] >= MAX_STEPS or state["escalate"]:
return "handoff"
return supervisor.choose(state) # claims | policy | answer
g = StateGraph(State)
g.add_node("claims", claims_agent) # sp-support-agent
g.add_node("policy", policy_agent) # sp-policy-reader
g.add_node("answer", compose)
g.add_node("handoff", to_human)
g.add_edge(START, "supervisor")
g.add_conditional_edges("supervisor", route)
g.add_edge("answer", END)
# Checkpointing = resumable runs + human-in-the-loop.
agent = g.compile(
checkpointer = checkpointer,
interrupt_before = ["write_claim_note"],
)Orchestration
Three things separate an agent you can run in production from a demo: the loop provably terminates, the state is a declared schema rather than a growing chat log, and there is a defined handoff when the agent is out of its depth.
Evaluation
This is the part that gets skipped, and it is the part that decides whether the project survives contact with real users. Agent evaluation is harder than model evaluation: there is no single right answer, the failure modes are subtle, and the thing you most need to measure is whether it was right for the right reason.
Built from actual traces and real expert answers, including the awkward cases and the ones the system should refuse. Versioned in Unity Catalog and grown every time production surprises us.
LLM judges and deterministic scorers, calibrated against human labels until their agreement is measured, not assumed. An uncalibrated judge is just a confident random number.
We score tool selection, retrieval quality and step count — because an agent that stumbles to a correct answer through six wrong tools will fail on the next question.
Evaluation runs in CI against pre-agreed floors for grounding, correctness and safety. A drop fails the build. No override without a named human.
Tracing, sampled scoring on live traffic, user feedback captured in the product, and cost and latency per request watched as first-class quality metrics.
Prompt injection through retrieved documents, permission escalation attempts, data exfiltration, jailbreaks and tool misuse — run as a suite, not as a one-off afternoon.
import mlflow
from mlflow.genai.scorers import (
Correctness, RetrievalGroundedness, Safety, Guidelines,
)
@mlflow.trace
def app(question: str):
return agent.invoke({"question": question, "steps": 0})
results = mlflow.genai.evaluate(
data = mlflow.genai.datasets.get_dataset(
"prod.agents.support_golden"),
predict_fn = app,
scorers = [
Correctness(),
RetrievalGroundedness(),
Safety(),
Guidelines(guidelines=[
"Cites a policy document for any entitlement claim.",
"Escalates rather than guessing a refund amount.",
]),
],
)
FLOORS = {"correctness": 0.90, "groundedness": 0.95,
"safety": 1.00, "guidelines": 0.95}
for metric, floor in FLOORS.items():
got = results.metrics[f"{metric}/mean"]
if got < floor:
raise SystemExit(f"FAIL {metric} {got:.3f} < {floor}")
# correctness 0.934 - groundedness 0.971 - safety 1.000 - PASSWhy this is the whole job
Once quality is a number, everything downstream gets easier. You can change a prompt, swap a model, add a tool or cut the critic step and know within minutes whether you improved the system or quietly broke it for a group of users nobody tested.
It also changes the conversation with your risk and compliance functions. They are rarely opposed to AI. They are opposed to things nobody can measure — and they are right.
Serving & operations
Logged and registered in Unity Catalog, deployed to Model Serving with traffic splitting so a new version proves itself on a slice first.
Every step, tool call, retrieved chunk, token count and latency captured in MLflow traces you can replay when someone reports a bad answer.
Input and output filtering, PII handling, per-tenant rate limits and a hard spend ceiling that fails safe rather than silently billing.
Thumbs-down and expert corrections flow back into the golden set, so every complaint becomes a permanent regression test.
Straight answers
Standardise on the tool boundary and the evaluation harness, not the orchestration library. We usually land on LangGraph for complex Python work, the Databricks Agent Framework where Unity Catalog governance is the binding constraint, and Microsoft Agent Framework for Azure and .NET estates. CrewAI earns its place in prototyping. Any of them is survivable; an unevaluated agent is not.
Treat retrieved content as untrusted input, never as instructions. Tools are allow-listed and scoped so the worst case of a successful injection is a bad answer rather than a bad action, consequential writes require human approval, and injection attempts are part of the standing red-team suite rather than a pre-launch afternoon.
We model it before we build: tokens per task, tool calls per task, and the latency budget. A critic loop roughly doubles cost; a supervisor with four specialists can be five times a single agent. Those numbers belong in the design decision, so we put them there and show you the cheaper topology.
Yes, and the first thing we do is instrument and evaluate it rather than rewrite it. Knowing where it actually fails usually reveals that the problem is retrieval quality or tool design, not the orchestration — which is a much cheaper fix than the rebuild you were expecting to pay for.
For anything that touches real systems, yes — and if it is not, we will say so. An agent is only as trustworthy as the smallest permission it can escalate, so the honest first engagement is sometimes data engineering, not agents.
Start here
Bring your current architecture, data volumes and the workloads that hurt. You leave with a target-state diagram on Databricks, the three changes that would cut the most cost or latency, and an honest answer on whether you need us at all.
No slide deck. No procurement process. One senior architect on the call.