AI Agents in Production

Production Challenges

Running agents locally is easy. Running in production is different:

• Observability: What is the agent doing? Why did it pick that tool?
• Cost: Every LLM call costs money. Multi-step agents = many calls
• Latency: Users don’t want to wait 30 seconds per response
• Reliability: What’s the agent failure rate? Recovery strategy?

Tracing & Observability

Trace Anatomy

Trace (1 user request)
├── Span: User input processing
├── Span: LLM call #1 (planning)
│   ├── Token usage: 500 input, 200 output
│   └── Duration: 1.2s
├── Span: Tool call: searchDocuments
│   └── Duration: 0.8s
├── Span: LLM call #2 (with tool results)
│   ├── Token usage: 1500 input, 300 output
│   └── Duration: 2.1s
└── Span: Response generation

Langfuse Integration

import { Langfuse } from "langfuse";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const langfuse = new Langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY!,
  secretKey: process.env.LANGFUSE_SECRET_KEY!,
  baseUrl: process.env.LANGFUSE_BASE_URL
});

async function tracedAgent(userMessage: string, userId: string) {
  const trace = langfuse.trace({
    name: "agent-response",
    userId,
    metadata: { source: "web-app" }
  });

  const planSpan = trace.span({ name: "planning" });

  const result = await generateText({
    model: openai("gpt-4o-mini"),
    tools: { /* ... */ },
    maxSteps: 5,
    prompt: userMessage,
    onStepFinish: (step) => {
      trace.span({
        name: step.toolCalls?.[0]?.toolName || "llm-generation",
        input: step.toolCalls?.[0]?.args,
        output: step.toolResults?.[0]?.result,
        metadata: { tokenUsage: step.usage, model: "gpt-4o-mini" }
      });
    }
  });

  planSpan.end();

  trace.generation({
    name: "final-response",
    input: userMessage,
    output: result.text,
    usage: {
      promptTokens: result.usage.promptTokens,
      completionTokens: result.usage.completionTokens
    }
  });

  trace.score({ name: "response-quality", value: 0.9 });
  await langfuse.shutdownAsync();
  return result.text;
}

Key Metrics

Metric	Description	Target
Latency P95	95th percentile response time	< 10s
Token Usage	Average tokens/request	Monitor trend
Error Rate	% failed requests	< 5%
Tool Call Success	% successful tool calls	> 95%
Cost per Request	Average cost/request	Monitor trend

Metrics Tracking

interface AgentMetrics {
  requestId: string;
  userId: string;
  timestamp: Date;
  totalLatencyMs: number;
  llmCalls: number;
  toolCalls: number;
  totalTokens: number;
  estimatedCost: number;
  success: boolean;
  errorType?: string;
}

async function logMetrics(metrics: AgentMetrics) {
  await supabase.from("agent_metrics").insert(metrics);
}

Cost Management

Token Cost Calculator

const PRICING = {
  "gpt-4o": { input: 2.50 / 1_000_000, output: 10.00 / 1_000_000 },
  "gpt-4o-mini": { input: 0.15 / 1_000_000, output: 0.60 / 1_000_000 }
};

function calculateCost(
  model: keyof typeof PRICING,
  inputTokens: number,
  outputTokens: number
): number {
  const pricing = PRICING[model];
  return inputTokens * pricing.input + outputTokens * pricing.output;
}

Cost Reduction Strategies

1. Model Routing

function selectModel(taskComplexity: "simple" | "medium" | "complex") {
  switch (taskComplexity) {
    case "simple": return openai("gpt-4o-mini");
    case "medium": return openai("gpt-4o-mini");
    case "complex": return openai("gpt-4o");
  }
}

2. Prompt Caching

// Use long system prompts — they get cached automatically
const cachedSystemPrompt = `
[Detailed system prompt]
This will be cached after the first call.
`;

3. Token Budget

const result = await generateText({
  model: openai("gpt-4o-mini"),
  maxTokens: 500,  // Limit output
  maxSteps: 3,     // Limit loop iterations
  prompt: userMessage
});

Error Handling & Recovery

async function resilientAgent(message: string) {
  const maxRetries = 3;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const result = await generateText({
        model: openai("gpt-4o-mini"),
        prompt: message,
        abortSignal: AbortSignal.timeout(30000)
      });
      return result.text;
    } catch (error) {
      console.error(`Attempt ${attempt} failed:`, error);

      if (attempt === maxRetries) {
        return "Sorry, I'm experiencing issues. Please try again later.";
      }

      await new Promise(r => setTimeout(r, 1000 * Math.pow(2, attempt)));
    }
  }
}

Summary

• Trace every agent step with Langfuse
• Monitor 5 key metrics: latency, tokens, errors, tool success, cost
• Cost management: model routing, prompt caching, token budgets
• Error recovery: retries with exponential backoff, fallback responses
• Production agents need observability first — you can’t fix what you can’t see