Customer Experience Agents in Production
A field white paper on running CX agents in production: the four-lens model, recurring agent patterns, eval pipelines and SLO-backed operations — grounded in four deployments: Vodafone, LATAM Airlines, Lyft, and Cloudflare.
EYwALINK · 15 Sep 2026 · 22-page PDF · ~10 min read · v1.1
In this paper
1. Executive Summary
Customer experience is the fastest-moving category in agentic AI — and the one where the gap between “the demo works” and “it is still working for that customer, in that conversation, right now” is widest. The moment a CX agent faces a real, unbounded, sometimes adversarial user, the problem stops being a model-accuracy problem and becomes a systems problem.
The organizing frame is a four-lens model — architecture, engineering, UX, and operations — that all have to evolve together, driven by the same production data. The four case studies in this paper are the evidence for every claim.
2. Why CX Agents Are a Different Beast
Unbounded, adversarial users
A customer can ask anything — including things the agent has no tool for, no policy answer for, and no business case to handle. The “out of scope” boundary is not a product decision you make up front; it is a data discovery problem you find by reading traces, not by spec.
The value is in doing
A CX agent that only answers questions is a Q&A machine. The value is in executing — activating an offer, disabling a service, updating a payment method, filing a claim — via structured action tags the frontend or backend can execute inside the conversation.
“Resolved” is end-to-end
A claim is not resolved when the agent says “I’ve escalated it.” It is resolved when it has collected evidence, run the tools, applied the decision, and explained the outcome — inside a defined time window.
The through-line: a CX agent is not a single “agent.” It is a system with four interacting layers — architecture, engineering, UX, and operations — that must evolve together.
3. The Four-Lens Model
The single most important idea in this paper: treat a CX agent as a system of four lenses, not a model you pointed at a FAQ. Each lens is a distinct discipline with its own artefacts, and all four are driven by the same production data.
| Lens | Question it answers | Key artefacts |
|---|---|---|
| Architecture | What is the topology, and how does it change? | Supervisor + specialist graph, router handoffs, subgraphs, retrieval layer |
| Engineering | How do we keep the agent correct as it grows? | Structured prompt templates, prompt hub, CI linting, eval pipeline, checkpoint store |
| UX | What does “resolved” mean for the customer? | Transactional action tags, end-to-end resolution, out-of-scope routing, knowledge capture |
| Operations | How do we know it is still working right now? | Per-agent dashboards, SLOs, alerting/paging, triage agent, daily eval loop |
The lenses are not sequential phases — they are coupled. A change in architecture (a new specialist) forces new engineering (prompt, tools, evals), changes UX (what the customer can now ask), and new operations (new SLOs, new dashboards). Production traces are the shared signal that moves all four at once.
4. What Makes a CX Agent Different
4.1 · The user is unbounded and adversarial
You cannot constrain the input. The out-of-scope boundary is found in traces: LATAM’s Concierge initially classified 13% of messages as out of scope — 95% of them were legitimate passenger needs. Adding a customer-care specialist dropped out-of-scope to 1% and lifted the return rate by 6%.
4.2 · The conversation is transactional
The value is in doing. The agent returns structured action tags — activate an offer, update a payment method — that the frontend or backend executes inside the conversation. Vodafone’s Super TOBI stops being a chatbot and becomes a transactional workflow.
4.3 · “Resolved” has a specific, end-to-end meaning
A driver damage claim is not resolved when the agent says “I’ve escalated it” — it is resolved when photos are collected, data pulled through tools, fraud signals applied, a decision made, and the outcome explained. Lyft’s 65% deflection and 35% AI resolution are the direct result of building to that definition.
4.4 · The conversation is a knowledge source
Every interaction contains intent, preferences, unmet needs, and future signals. LATAM’s Compass turns the unstructured corpus of conversations into structured knowledge: entities and relationships mapped against a domain ontology and written to a knowledge graph. The ontology — not the model — is the proprietary asset.
4.5 · Real-time latency and cost matter operationally
A CX agent in a support queue has a p95 latency budget; a driver in a fare dispute will not wait 30 seconds. Every agent in production should have a per-agent dashboard tracking run volume, error rates, p50/p95 latency, token usage, tool-call success, and judge scores — with an operational alert that fires on a real SLO.
4.6 · The user can also be the operator
The same discipline applies when the user is configuring their own environment instead of asking a company for help. Cloudflare’s agent lets a user set up their own Workers, DNS, and network configuration conversationally on a plain white page — no dashboard, just a chat. The out-of-scope boundary, safety gates, and eval discipline are the same; what changes is the trust model: the agent acts on the user’s own resources, with the user’s own authority.
5. The Hard Challenges
The hard parts are not “get the LLM to answer correctly.” They are operational and organizational.
Evals are the new test suite — and harder than unit tests
- ▸ Generic scores (“helpfulness: 4/5”) are non-actionable. Binary pass/fail judgments, calibrated against human labels, are the minimum bar.
- ▸ Offline simulation is necessary but insufficient: a simulated user that is too polite will let an agent pass offline that fails in production. Ground personas in real customer verbatims.
- ▸ The eval pipeline itself is a product: a 5–10% production rollout, trace sampling, and LLM-as-a-Judge evaluators running automatically on production traces.
- ▸ Daily cadence matters: running evals once is a release gate; running them daily on a rolling window of production traces is an operational habit.
Prompt quality is the real bottleneck, not the platform
- ▸ When Lyft opened agent building to non-engineers, the hardest part was not tool bindings or state management — it was prompt quality: happy-path prompts that fell apart on edge cases, vague tone sections, and missing out-of-scope definitions.
- ▸ The fix is a structured five-component prompt template, paired with a CI linting pipeline (static + LLM-powered rules) that blocks every merge on violation.
- ▸ Treat prompts like product specs, not code comments. The more explicit the prompt, the more consistent the agent.
The five-component prompt template
- 1 · IdentityWho is this agent, what user type, what topic area.
- 2 · Primary objectiveConcrete verbs — not vague “help” or “handle”.
- 3 · ScopeBoth in-scope and out-of-scope, with explicit routing actions for out-of-scope.
- 4 · Phased workflowNumbered steps with entry conditions, branching for every if/else, and a terminal action for every phase.
- 5 · Content guidelinesConcrete do/don’t rules with example phrases, not abstract principles.
Observability is a day-one requirement, not a postmortem
- ▸ Trace every invocation across dev, staging, and production: which nodes ran, what the LLM saw, which tools were called, token usage, latency at every step.
- ▸ Enrich traces with custom metadata (user type, agent name, intent, conversation ID) and give every agent in production its own cloned dashboard.
- ▸ Wire alerts to a paging system: error rate above 5% or p95 latency above 10 s over a 15-minute window pages on-call automatically.
- ▸ Build a triage agent: a second agent that pulls failing traces, groups related issues, and opens tickets. Humans design the fix; the triage agent sorts the noise.
The architecture is not finished at launch
- ▸ The supervisor pattern, the subgraph structure, even which agent formats the final response — all of it gets tuned after observing real conversations.
- ▸ Ship a supervisor that can hand off, return control, and re-route. The specialist set will grow, shrink, and merge based on what production traces show.
6. Agent Patterns: The Recurring Shapes
Across the three support deployments, the same core structure repeats: a Supervisor or meta-agent owns the conversation, specialist agents handle discrete use cases, and tools / APIs do the transactional work. The specific patterns within that structure are what matter.
| Pattern | What it does | Where it shows up |
|---|---|---|
| Router with stateful handoffs | Separate router instances per user type; a subagent returns control to the parent on a domain shift — Command(goto=…, graph=PARENT) | Lyft (riders vs. drivers) |
| Specialized vs. configurable split | MLE-hand-built agents for complex, high-stakes flows; JSON-config + prompt-hub agents for the self-serve layer | Lyft |
| Procedural subgraph (“LLM Compiler”) | Store procedures in a graph DB; the LLM picks which procedure and in what order; the procedure stays deterministic | Vodafone (Neo4j) |
| Hybrid retrieval (vector + graph) | Vector search finds chunks; the graph layer captures the relationships between them (“this supersedes that”) | Knowledge-intensive CX |
| Centralized formatting | Move response formatting from each specialist back to the supervisor for consistency and cost | LATAM (15% cost cut) |
The specialized vs. configurable split is the architectural enabler for self-service: new agents are defined by writing a prompt and a JSON config, not by engineering a new state graph. The platform handles graph construction, tool execution, checkpointing, tracing, and safety.
7. The Architecture That Holds It Together
7.1 State persistence and checkpointing
For multi-turn conversations the checkpoint store is not a detail — it is your ability to answer “what exactly
did the agent see and decide at step 7?” Persist the full graph state, execution metadata, and parent
checkpoint references after every node execution. That enables conversation replay, debugging, and state
inspection in production. The interface is LangGraph’s BaseCheckpointSaver; the implementation is
storage-engine-agnostic — DynamoDB for a serverless setup, PostgreSQL for on-prem (Lyft uses
DynamoDBSaver).
7.2 The ontology layer
If your agent answers questions about policies, products, or procedures, the retrieval layer should not be pure vector search. Build an ontology-driven pipeline: parse unstructured sources (conversations, interviews, legal docs) → map entities and relationships against a domain ontology → write to a knowledge graph → run hybrid retrieval over the graph at query time. The ontology encodes the business’s own definitions — that is the durable asset; the model is a tool.
7.3 The prompt hub as a deployment concern
A prompt hub is not just version control for prompts — it is the deployment concern between domain-expert knowledge and running agent behavior. It should support structured templates with required fields, prompt linting (static + LLM-powered) before activation, prompt A/B evaluation with human reviewers, and continuous scoring on production traces with automatic degradation alerts. The prompt hub is where prompt quality becomes an engineering discipline rather than an individual’s craft.
7.4 The eval pipeline as a product gate
Before any agent rolls out to 100% of traffic, it must pass a three-stage process:
- 1.Small production rollout (5–10%) — the agent serves real traffic at low volume.
- 2.Sample production traces — real conversations captured as evaluation datasets.
- 3.Run LLM-as-a-Judge evaluators — a shared judge template from the prompt hub, extended with agent-specific metrics; sampling tapers as confidence grows.
8. The Day-to-Day: Evaluation and Operations
8.1 · The daily eval loop
Collect traces overnight, run the evaluators, generate structured output — scores, explanations, violated guidelines. Business stakeholders review daily and feed prompt adjustments back. The target is a 90% correctness threshold, not “it looks fine.”
8.2 · The simulation infrastructure
Before a new agent ships, test it against synthetic conversations with mocked tool responses: personas grounded in real verbatims (refund seekers, AI skeptics, multilingual customers), edge cases production has revealed, and pairwise A/B comparison of prompt revisions with human reviewers. Catch regressions before a single real customer sees the output.
8.3 · The triage agent
A second agent pulls failing traces from the observability platform, groups related issues, and opens tickets automatically. It doesn’t fix anything — it sorts the noise so the human engineer spends time on the fix, not the triage.
8.4 · SLOs and alerting
| Signal | Threshold | Action |
|---|---|---|
| Error rate | sustained > 5% over 15 min | page on-call |
| p95 latency | sustained > 10 s over 15 min | page on-call |
| Tool-call success rate | drop below threshold | upstream API problem, not the LLM |
| Judge-score trend | declining over N days | prompt review (not a page) |
The goal is not a pretty Grafana screen. It is a structured pipeline that turns bad interactions into better prompts, workflows, and models — and pokes the on-call when something crosses a line.
8.5 · Agent-level telemetry
The discipline above is not CX-specific. Cloudflare’s 2026 agent-tracing launch names the same failure mode from the infrastructure side: “An agent can return HTTP 200 and still fail. It may choose the wrong tool, pass stale context to a subagent, or spend tokens in a retry loop.” Traditional application telemetry shows the HTTP request or the database query — but not the agent behavior that caused it. Cloudflare’s answer is agent-aware spans nested inside the Workers infrastructure spans: model calls, tool execution, subagent handoffs, and token usage as first-class trace nodes, exportable to any OpenTelemetry-compatible destination. Two views: replay a session (the full conversation) and check the trace (the execution waterfall — where time went, which subagent did the work, which D1/KV calls fired). The same SLO discipline as the Lyft case — but the telemetry is agent-level, not HTTP-level.
9. Case Studies: The Evidence
Vodafone / Fastweb
Super TOBISupervisor + use-case agents, each with a defined API set, following the “LLM Compiler” pattern: the Super Agent routes to either a procedural subgraph (troubleshooting procedures in Neo4j) or a hybrid vector + graph retrieval pipeline.
Structured action tags for transactional operations — activating an offer, disabling a service, updating a payment method. The agent is a workflow, not a Q&A machine.
Daily automated evaluation with the Evaluators SDK: collect traces overnight, run evaluators, generate structured output (scores, explanations, violated guidelines). 90% correctness target.
Daily eval cadence; the structured output is designed for non-technical stakeholders — they can read it and act without understanding the underlying graph.
LATAM Airlines
Concierge + CompassTool-per-agent layout with a supervisor delegating to specialists for flights, bookings, destinations, and customer care. Response formatting moved from each specialist back to the supervisor: a 15% cost reduction caught only by reading individual traces.
13% of messages initially classified out of scope; traces revealed 95% were legitimate passenger needs. A customer-care specialist dropped out-of-scope to 1%; return rate up 6%.
Compass: ontology-driven knowledge extraction — parse unstructured sources, map entities and relationships against a domain ontology, write to a knowledge graph. The ontology is the proprietary asset.
Trace-level observability revealed both the duplicate-formatting cost issue and the out-of-scope discovery; a triage agent pulls failing traces and groups related issues.
Lyft
Self-Serve Agent PlatformRouter pattern with separate instances for riders and drivers; Command(goto=…, graph=Command.PARENT) for mid-conversation handoffs. The specialized vs. configurable split enables self-service: new agents are a prompt plus a JSON config, not a new state graph.
Five-component structured prompt template; Git-backed prompt-linting CI (static + LLM-powered, blocking merges); three-stage eval pipeline with binary judge outputs; DynamoDBSaver for state persistence.
65% deflection, 35% AI resolution. Missing out-of-scope definitions were the most consistent prompt failure mode — every prompt now requires explicit out-of-scope routing actions.
Per-agent dashboards; PagerDuty integration pages on-call when error rate > 5% or p95 > 10 s over 15 minutes.
Agent development from ~6 months to ~2 weeks; 100% of production agents have automated eval pipelines; hallucination and contradiction rates down 20% with guardrails built from eval metrics; AI resolution up 16%.
Cloudflare
Self-Serve Configuration AgentA surface agent that lets a user configure their own Cloudflare environment — Workers, DNS, network settings — conversationally on a plain “white page.” The user is the operator, not a support caller. The agent translates intent to config actions and applies them on the user’s own resources.
No dashboard; the interface is a chat. The “white page” is deliberate: the agent is the configuration surface. Structured action tags apply — the agent returns the config changes it made, not just a text response.
Agent tracing: agent-level spans (model calls, tool execution, subagent handoffs, token usage) nested inside the Workers infrastructure spans. OpenTelemetry-compliant, exportable to any OTLP destination — the same data that helps debug the agent powers evals, analytics, and cost reporting.
“An agent can return HTTP 200 and still fail.” Agent-aware tracing with two views: replay a session (full conversation) and check the trace (the execution waterfall). The same SLO discipline as the Lyft case — but the telemetry is agent-level, not HTTP-level.
The “user as operator” archetype (§4.6). Same disciplines as Lyft’s configurable split, applied to user-facing configuration. The agent acts on the user’s own resources with the user’s own authority, so the safety gates are scoped to the user’s environment, not a company’s data.
10. Design Principles: What This Means for Builders
- 1
The architecture is a living thing. Ship a supervisor that can hand off, return control, and re-route — the specialist set will grow, shrink, and merge with the data.
- 2
Evals are the product gate. Use binary pass/fail judgments, narrow behavior-specific rubrics, judges calibrated against human labels, offline simulation as a launch gate, and online annotation as the continuous loop.
- 3
The user’s “out of scope” is your coverage gap. The out-of-scope rate is a coverage metric, not a user-behavior metric.
- 4
Observability is the feedback channel — wire it into alerting: error-rate thresholds, p95 SLOs, tool-call success rates, and a paged on-call when something crosses a line.
- 5
The conversation is a knowledge source. Agents are not just answerers; they are sensors. The ontology is the durable asset; the model is a tool.
- 6
Prompts are product specs, not code comments. Build the structured template, build the linting pipeline, and make domain experts pass through CI before a prompt reaches production.
- 7
The user can be the operator. When the user configures their own environment (Cloudflare’s white-page agent), the disciplines are the same — structured prompts, evals, agent-level tracing, SLOs — but the trust model shifts: the agent acts on the user’s own resources, so the out-of-scope boundary and the safety gates are scoped to the user’s environment, not a company’s data. The architecture is the same; the permission model is different.
11. How EYwALINK Closes These Gaps
EYwALINK’s four strategic pillars are a problem-first decomposition of exactly what this white paper describes. Each pillar corresponds to a section; the product portfolio closes the specific gaps the paper names.
| Pillar | White-paper anchor | What it adds |
|---|---|---|
| Pillar 1 · AI Strategy & Architecture | §5.4 “the architecture is not finished” + §6 patterns | Executable EA maps the TOGAF target state to the supervisor/specialist topology and keeps it a living artifact that evolves with production data — not a diagram that goes stale at launch. |
| Pillar 2 · AI Safety & Privacy | §4.1 “unbounded and adversarial” + prompt linting | Safety gates, out-of-scope definitions, and prompt-injection detection in the linting pipeline are compliance requirements for regulated CX (telco, aviation, healthcare). No data leaves client infrastructure: ontology, eval judges, and knowledge graph all run client-side. |
| Pillar 3 · Rapid AI-Led Application Development | §9 Lyft “6 months → 2 weeks” + the configurable split | “Rapid” means structured prompt templates with CI gates, not improvised prompts. Domain experts write the prompt; the platform handles graph construction, tool binding, checkpointing, and tracing. |
| Pillar 4 · AI Observability & Operations | §8 Evaluation and Operations | Model-health tracking, automated performance-regression detection, agent-failure detection, SLA-backed incident response, and a fractional AI-operations team that is always on and always accountable. |
The product portfolio, mapped to the gaps
§8.4 per-agent dashboards + PagerDuty wired to SLOs — self-hosted on client infrastructure; no data leaves the perimeter.
§8.2 simulation infrastructure + §7.4 three-stage eval pipeline — synthetic-conversation testing with mocked tool responses before a real customer sees the output.
§8.3 the triage agent plus the ADLC iterate loop — the agent that reads failing traces and opens tickets is a Loop Factory instance.
§10.1 operationalized — the target-state diagram is the starting state of a continuous cycle where production data feeds back into the architecture.
§4.6 / §10.7 “user as operator” (Cloudflare’s white page) — Pillar 3 (rapid AI-led development) builds these; Pillar 4 (observability) is what makes the user trust the agent acting on their own resources; the white-page UX is the minimal surface a configuration agent needs. The user owns the resources, so Pillar 2 is the more relevant constraint.
The reference stack (LangGraph + LangSmith + PagerDuty) is a proprietary platform the client does not own. The operational discipline is the same either way — what changes when you own the infrastructure is that the out-of-scope boundary, the ontology, the trace data, and the eval judges are all client assets rather than platform dependencies. The zero-lock-in, private-AI position is not a compromise; it is the prerequisite for AI safety and privacy in a regulated CX environment.
12. References
- · How Lyft Built a Self-Serve AI Agent Platform for Customer Support with LangGraph and LangSmith — Akshay Sharma, Lyft SCX Data Science & MLE team, 27 May 2026
- · The Agent Development Lifecycle — LangChain, 9 May 2026
- · Customer Experience Agents in Production: Lessons from Lyft, Vodafone, and LATAM Airlines — LangChain field notes
- · Introducing: Cloudflare Agents — Nevi Shah, Matt Simpson, Fred Schott, Cloudflare, Aug 2026
- · What is an AI agent? — Cloudflare Learning
- · Agent setup — Cloudflare Developer Docs — Cloudflare
All numbers and quotes trace to the sources above.
The Four Lenses Turn a CX Agent Into an Operating System
The reference stack is a platform you rent. The discipline — structured prompts, calibrated binary evals, daily eval cadence, SLO-backed alerting, a triage agent — is the same either way.
Get started
The primary source behind this paper: lessons from Lyft, Vodafone, and LATAM Airlines.
Walk through how a target-state diagram stays a living artifact as production data feeds back into it.
We'll wire the four lenses onto your CX agent system — dashboards, SLOs, and the triage agent in the first weeks.
Want the four lenses on your CX agent system? Contact us — we'll map a 2-week eval pipeline + SLO wiring on your stack.