07. Deploy, Rollback & Observability — Vibe Coding Series

Ten minutes after deploying, error rate jumped 300% — and I had no idea what the AI had changed in that codebase.

Not because the code was bad. Not because tests were missing. But because I had no observability, no rollback plan, and no clear picture of exactly what had gone to production. AI writes code fast, but when something breaks, I was staring at a dashboard with no idea where to start.

That was the last time I deployed without a plan.

---

Context

In article 06, the code passed automated QA, went through AI security review, and got a human final check before merging to main. This article is the last step: shipping to production in a controlled way — and knowing exactly what to do when things don’t go as planned.

---

Deploy strategy for AI-generated code

AI-generated code has a particular characteristic: it tends to work well in test environments but can behave unexpectedly in production with real traffic. Not because AI writes worse code — but because AI has no context about your production environment, real user edge cases, or actual load patterns.

The right deploy strategy is gradual rollout.

Feature flags: controlling the rollout

Feature flags let you deploy code to production but only enable it for a small group of users first. If something goes wrong, flip the flag off — no code rollback needed.

For TaskFlow, I use a simple implementation:

// lib/feature-flags.ts
const FLAGS: Record<string, { enabled: boolean; rolloutPercent: number }> = {
  "auth-v2": { enabled: true, rolloutPercent: 10 }, // 10% of users
  "new-dashboard": { enabled: false, rolloutPercent: 0 },
};

export function isFeatureEnabled(flagName: string, userId: string): boolean {
  const flag = FLAGS[flagName];
  if (!flag?.enabled) return false;

  // Deterministic rollout based on userId
  const hash = userId.split("").reduce((acc, c) => acc + c.charCodeAt(0), 0);
  return (hash % 100) < flag.rolloutPercent;
}

Using it in a route handler:

// routes/auth.ts
if (isFeatureEnabled("auth-v2", req.user.id)) {
  return authV2Handler(req, res);
}
return authV1Handler(req, res);

When the rollout looks stable (error rate flat, latency normal), increase rolloutPercent gradually: 10% → 25% → 50% → 100%. Once you hit 100%, remove the flag and the old code.

Canary deployment

If you’re running containers or Kubernetes, canary deployment is the next level: deploy the new version to 10% of pods, keep 90% on the old version. Traffic splits automatically.

With simpler setups (Railway, Render, etc.), feature flags already achieve the same goal.

AI-generated deployment checklist

Before every deploy, I use AI to generate a checklist from the diff:

I'm about to deploy these changes to production. Create a deployment checklist:

Changes:
- Added auth-v2 with JWT refresh token rotation
- Added refresh_tokens table to the database
- Added rate limiting on /auth/login

Checklist should cover:
1. Any database migrations to run? In what order?
2. New environment variables to set?
3. Which services need restarting?
4. Rollback plan if something goes wrong?
5. Which metrics to monitor in the first 30 minutes?

Takes 2 minutes to generate, saves a lot of headaches.

---

Fast rollback

When something breaks, every minute of downtime has a cost. The rollback plan needs to be ready before the deploy, not improvised during an incident.

Git tag strategy

Tag every deploy so you have a clear rollback point:

# Before deploying
git tag -a deploy-2024-01-15-v1.2.3 -m "Deploy: auth-v2 feature flag at 10%"
git push origin deploy-2024-01-15-v1.2.3

# Deploy to production
npm run deploy:production

# If rollback is needed
git checkout deploy-2024-01-14-v1.2.2
npm run deploy:production

Convention I use: deploy-{date}-{semver}. One glance at the tag tells you which deploy, when.

For database migrations, rollback is more complex — you need a down migration script and you need to test it before running the up.

Automated rollback trigger

With real production traffic, I set up an alert to automatically rollback when error rate crosses a threshold:

// monitoring/auto-rollback.ts
const ROLLBACK_THRESHOLD = {
  errorRatePercent: 5,    // > 5% of requests failing
  p99LatencyMs: 2000,     // p99 latency > 2s
  windowMinutes: 5,       // measured over 5 minutes
};

async function checkAndRollback() {
  const metrics = await getMetrics(ROLLBACK_THRESHOLD.windowMinutes);

  if (metrics.errorRate > ROLLBACK_THRESHOLD.errorRatePercent) {
    await notifySlack(`🚨 Error rate ${metrics.errorRate}% — triggering rollback`);
    await rollbackToLastStableTag();
  }
}

Automated rollback isn’t always right — sometimes a spike is unusual traffic, not a bug. But it’s an important safety net when you’re not watching the dashboard around the clock.

---

Observability

“AI code has fewer bugs” is one of the most dangerous assumptions you can make. AI code may have fewer syntax errors, but logic bugs and edge cases still happen — and when they happen in production, you need to know immediately.

Logging strategy

AI-generated code tends to be under-logged — AI writes code to run correctly, not to be debugged. My rule: whenever AI writes an important function, I ask it to add structured logs.

// Instead of letting AI write this:
async function loginUser(email: string, password: string) {
  const user = await db.users.findByEmail(email);
  if (!user || !verifyPassword(password, user.passwordHash)) {
    throw new Error("Invalid credentials");
  }
  return generateTokens(user.id);
}

// Ask AI to add logging:
async function loginUser(email: string, password: string) {
  const logger = createLogger("auth.login");

  const user = await db.users.findByEmail(email);
  if (!user) {
    logger.warn("Login attempt for non-existent email", { email });
    throw new UnauthorizedError("Invalid credentials");
  }

  if (!verifyPassword(password, user.passwordHash)) {
    logger.warn("Failed login attempt", { userId: user.id, email });
    throw new UnauthorizedError("Invalid credentials");
  }

  logger.info("Successful login", { userId: user.id });
  return generateTokens(user.id);
}

Structured logs (JSON format) are much easier to query than plain text when debugging a production issue.

Tracing

For microservices or apps with multiple layers, distributed tracing lets you follow a request through every service and see where time is being spent.

For TaskFlow (a small monolith), I use OpenTelemetry to trace requests from the HTTP handler down to the database:

// Wrap database call with a span
const span = tracer.startSpan("db.users.findByEmail");
try {
  const user = await db.users.findByEmail(email);
  span.setStatus({ code: SpanStatusCode.OK });
  return user;
} catch (err) {
  span.recordException(err);
  span.setStatus({ code: SpanStatusCode.ERROR });
  throw err;
} finally {
  span.end();
}

When there’s a latency spike, the trace shows you immediately: is it the database query or the application logic?

Alerts: AI-assisted anomaly detection

Instead of manually setting thresholds for every metric, I use AI to analyze patterns and suggest alerts:

Here are my app's metrics over the past 7 days:
- Error rate: avg 0.3%, peak 1.2%
- p99 latency: avg 180ms, peak 450ms
- Requests/min: avg 120, peak 380

Suggest reasonable alert thresholds for:
1. Error rate (page immediately vs. investigate)
2. Latency (user-impacting vs. warning)
3. Traffic anomaly (unusual spike)

AI will suggest thresholds based on your actual baseline, not arbitrary numbers.

---

Retrospective

Deploying isn’t the finish line. Retrospective is the most important step for making the framework better over time.

Evaluating the framework after each project

After TaskFlow’s auth feature was stable in production, I spent 30 minutes answering:

• Which phase took the most time? Why?
• Where did AI perform best? Worst?
• Was there a step I skipped that I later paid for?
• If I did this again, what would I change?

No need to write an essay — 5–10 bullet points is enough. The important thing is doing it while the memory is fresh.

Updating AI_CONTEXT.md

AI_CONTEXT.md is the context file I feed to the AI agent at the start of each session. After each project, I update it:

## Lessons learned (updated 2024-01-15)

### What works well
- Feature flag pattern for AI-generated features: safe rollout, easy rollback
- Structured logging from the start: saves significant debug time
- Git tag before every deploy: rollback in < 5 minutes

### What to improve
- AI often skips database indexes — need explicit prompt
- Migration scripts need more careful review before deploy
- Alert thresholds should be set right after first deploy, not after an incident

This file becomes institutional memory — next time you start a new project, the AI agent has context about what you’ve already learned.

Lessons for next time

A pattern I find useful: after every incident (even small ones), write a short mini post-mortem:

• Incident: what happened
• Root cause: why it happened
• Detection: how it was found (and how long it took)
• Resolution: how it was fixed
• Prevention: what to do so it doesn’t happen again

Doesn’t need to be formal — 5 sentences is enough. But this pattern stops you from making the same mistake twice.

---

Real-world example: Deploying TaskFlow’s auth feature

After the auth feature passed QA in article 06, here’s the deploy process I went through:

Step 1: Prepare

# Tag current stable version
git tag -a deploy-2024-01-14-v1.1.0 -m "Stable before auth-v2 deploy"
git push origin deploy-2024-01-14-v1.1.0

# Run migration on staging first
npm run db:migrate:staging
npm run test:integration:staging

Step 2: Deploy with feature flag off

Deploy the code to production with auth-v2 flag at rolloutPercent: 0. No users hit the new code, but it’s ready.

Step 3: Enable for 5% of users

"auth-v2": { enabled: true, rolloutPercent: 5 }

Monitor for 30 minutes: error rate, latency, login success rate. All normal.

Step 4: Ramp up gradually

5% → 25% → 50% → 100% over 2 days. Monitor 30–60 minutes at each step.

Result: Zero downtime, zero rollbacks needed. If something had gone wrong at any step, I just needed to set rolloutPercent: 0 — no code revert, no emergency deploy.

---

Pitfalls

Deploying without a rollback plan. “We’ll fix it if something breaks” is not a plan. The rollback plan needs to exist before the deploy, not after the incident. Git tagging takes 30 seconds — there’s no excuse to skip it.

Skipping observability because “AI code has fewer bugs”. AI code may have fewer syntax errors, but logic bugs, race conditions, and edge cases still happen. Without observability, you’ll find out about problems when users complain — not when metrics spike. That’s the difference between 5 minutes and 5 hours of downtime.

---

Looking back at the full framework

This is the last article in the series. Looking back at all 8 phases:

• Phase 0 (Foundation): Environment setup, project memory, glossary
• Phase 1 (Requirements): Analyze requirements, lock in architecture with AI as sparring partner
• Phase 2 (Specs & API): Contract first — spec and API before code
• Phase 2.5 (Design System): Design tokens and atomic components for AI consistency
• Phase 3 (Task Planning): Right granularity, test strategy before coding
• Phase 4 (Execution): Single agent, multi-agent, HITL checkpoints
• Phase 5 (QA & Review): Automated QA, AI-assisted review, human final check
• Phase 6 (Deploy): Feature flags, rollback plan, observability, retrospective

This framework isn’t a silver bullet. It’s a structured approach to working with AI agents — reducing risk, increasing predictability, and helping you learn faster from each project.

Advice for each reader profile

Solo developer: You don’t need to apply all 8 phases right away. Start with Phase 1 (clear requirements) and Phase 4 (HITL checkpoints) — these have the highest ROI. Add other phases as you feel the need.

Small team (2–5 people): Phase 5 (QA & Review) and Phase 6 (Deploy) matter most when multiple people are using AI. You need shared agreement on review process and deploy strategy to avoid conflicts.

Senior developer: This framework might feel like “too much process” if you’re used to moving fast. Think of it as a checklist — not every step needs to be done in full, but you should know which steps you’re skipping and why.

The most important thing I’ve learned after using AI agents across multiple projects: AI performs best when you know what you want. This framework isn’t about controlling AI — it’s about helping you know what you want before you start.

---

← Article 06: QA, Code Review & PR Workflow