The Debugging Story: Customers Silently Downgraded

The Ticket

A handful of support tickets land, all with the same shape:

I resubscribed to Pro last week, my card was charged, but the dashboard still shows me on the Free plan and I just hit the review limit again.

Checking Stripe: the charge succeeded, the subscription shows status: active. Checking the businesses table: subscription_tier is free and subscription_status is canceled -- the old subscription's terminal state, not the new one. It is not every resubscription, either -- most go through fine. It only hits customers who canceled and then resubscribed within the same billing relationship: same Stripe customer, new subscription object. Brand new customers checking out for the first time are never affected. That asymmetry is the first real clue -- whatever is wrong only shows up when there is a history on the account, not on a clean one.

Step 1: Try to Reproduce It

The obvious first move is the Stripe CLI: stripe trigger customer.subscription.updated. One event, fired in isolation, always reconciles correctly. No reproduction. That single negative result is informative rather than a dead end -- if one event in isolation always works, the bug is not in how any individual event type is handled. It points at a sequence problem: something about the order or timing of multiple events for the same customer, not a logic bug in any one handler.

Step 2: Read the Actual Code Path

app/api/billing/webhook/route.ts handles checkout.session.completed, customer.subscription.created, customer.subscription.updated, and customer.subscription.deleted, and every one of them funnels through a shared syncSubscription():

async function syncSubscription(subscription: Stripe.Subscription) {
  const customerId =
    typeof subscription.customer === "string"
      ? subscription.customer
      : subscription.customer.id;

  const [business] = await db
    .select()
    .from(businesses)
    .where(eq(businesses.stripeCustomerId, customerId))
    .limit(1);

  if (!business) return;

  // original version -- no guard at all
  await db
    .update(businesses)
    .set({
      subscriptionTier: tierFromStatus(subscription.status),
      subscriptionStatus: subscription.status,
      stripeSubscriptionId: subscription.id,
      updatedAt: new Date(),
    })
    .where(eq(businesses.id, business.id));
}

That is the smell. The lookup matches on customer, not subscription, and the write unconditionally overwrites subscription_tier, subscription_status, and stripe_subscription_id with whatever the incoming event says -- no check that the incoming event's subscription is actually the business's current one. Any event for any subscription that customer has ever had gets applied, last write wins, with "last" meaning "last to arrive at this endpoint," not "last to actually happen."

Step 3: Read Stripe's Docs Instead of Assuming

The instinct at this point is to assume Stripe delivers events in the order they happened. Stripe's own webhook documentation says otherwise, explicitly: delivery order is not guaranteed. Retries -- triggered by a 5xx or timeout from the receiving endpoint -- network delays, and Stripe's own internal queuing can all reorder events relative to when they actually fired. This single fact reframes the whole bug: it is not a logic error in any handler, it is an assumption (events arrive in causal order) that the code never stated but depended on everywhere.

Step 4: Reconstruct the Real Sequence

Pulling the event log for one affected customer directly from the Stripe dashboard shows the actual sequence of deliveries to the webhook -- not the order the underlying events originally fired in:

1. customer.subscription.created for sub_new, status active -- the resubscription. Processed fine; the business correctly shows Pro at this point.
2. customer.subscription.updated for sub_old, status canceled -- this was supposed to have been delivered before step 1, as the tail end of the original cancellation. But the endpoint returned a timeout on its first delivery attempt, and Stripe's retry landed after the new subscription's activation event.

Step 2 overwrites the business back to free / canceled, keyed by sub_old's id -- because syncSubscription() has no concept of "is this subscription id still the one I should be listening to." A retried, stale event for an already-superseded subscription silently clobbers newer, correct state.

Root Cause

syncSubscription() keys all writes off stripeCustomerId alone and applies every event unconditionally, with no check against the subscription id already on file or whether that stored subscription is even still "live." Combined with Stripe's documented lack of delivery-order guarantees, this is a stale-write / event-ordering bug -- not a one-off mistake in handling one event type. Every event type funnels through the same unguarded write, so every event type was equally exposed.

The Fix

The merge logic moves out of the webhook route entirely, into a pure function: lib/billing/reconcile.ts exports reconcileSubscriptionUpdate(business, subscription), extracted specifically so it could be unit tested without standing up a live database or signing a fake Stripe payload. It adds exactly one guard before applying a patch:

const TERMINAL_STATUSES: Business["subscriptionStatus"][] = [
  "canceled",
  "incomplete_expired",
  "unpaid",
  "none",
];

export function reconcileSubscriptionUpdate(
  business: Business,
  subscription: Stripe.Subscription,
): SubscriptionPatch | null {
  const hasDifferentSubscriptionOnFile =
    business.stripeSubscriptionId !== null &&
    business.stripeSubscriptionId !== subscription.id;
  const storedSubscriptionStillCurrent = !TERMINAL_STATUSES.includes(
    business.subscriptionStatus,
  );

  const isStale = hasDifferentSubscriptionOnFile && storedSubscriptionStillCurrent;

  if (isStale) {
    return null;
  }

  return {
    subscriptionTier: tierFromStatus(subscription.status),
    subscriptionStatus: subscription.status,
    stripeSubscriptionId: subscription.id,
  };
}

The rule: if the incoming event's subscription id differs from what is stored, and the stored subscription's status is still non-terminal (active, trialing, past_due, incomplete -- i.e. it still looks "current") -- the event is treated as stale and ignored. If the stored subscription has already reached a terminal status, or there is nothing on file yet, a different subscription id is a legitimate resubscription and the patch applies normally. That distinction -- old event for a subscription we have moved past, versus a genuinely new subscription -- is exactly what the business's own stored status tells you, even though both cases produce "a different subscription id than what's on file."

The First Fix Attempt Was Wrong

This is the part worth sitting with. The first version of the guard was simpler: reject any event whose subscription id did not match what was stored, full stop. It looked reasonable, and it would have fixed the original bug. But adding the resubscription test case before shipping caught that it broke a different, equally real scenario: a genuine resubscription where the business's stored subscription id is still the old, already-canceled one. Rejecting on id mismatch alone treats every resubscription as stale -- which trades a silent-downgrade bug for a silent-upgrade-never-applies bug. Same symptom from the customer's side (I paid, my account is wrong), opposite direction, caught only because a test was written for the case the first patch did not yet handle. The fix that shipped checks the stored subscription's status, not just whether the id matches, specifically because of what that test exposed.

The Tests, in the Order They Were Added

tests/reconcile.test.ts reads like the investigation itself, because it was written in the order the investigation happened:

1. A normal active-subscription event upgrades a free business -- sanity check, passed before and after the fix.
2. The bug, reproduced directly: a stale canceled event for an old subscription id must not overwrite a business that has since activated a newer subscription. Against the original code this failed with expected null, received { subscriptionTier: 'free', ... } -- and passes after the fix.
3. A non-stale event, matching the business's current subscription id, still applies normally -- confirms the guard does not break the common case.
4. The edge case the first fix attempt missed: a genuine resubscription where the stored id is the old, canceled one. The id-only guard rejected this as stale; the status-aware guard does not.
5. A first-ever subscription for a business with nothing on file yet (stripeSubscriptionId: null) still applies normally -- there is nothing for it to be stale relative to.

it("does not let a stale event for an old, canceled subscription downgrade
   a business that has since resubscribed", () => {
  const business = makeBusiness({
    subscriptionTier: "pro",
    subscriptionStatus: "active",
    stripeSubscriptionId: "sub_new",
  });
  const staleEvent = makeSubscription({ id: "sub_old", status: "canceled" });

  const patch = reconcileSubscriptionUpdate(business, staleEvent);

  expect(patch).toBeNull();
});

All 9 unit tests in the suite pass; npx tsc --noEmit and npm run build are both clean. The webhook route itself (app/api/billing/webhook/route.ts) was refactored to call reconcileSubscriptionUpdate() instead of inlining the merge logic, with no behavior change beyond the new guard -- the route's job is now just "verify the signature, fetch the business, ask the pure function what to do, apply the patch if there is one."

What This Doesn't Fix

Worth stating plainly, the way the original investigation did, rather than letting the fix look more complete than it is. This guard handles ordering between webhook deliveries for the same business, using data already on the row. It does not:

• Deduplicate truly identical redelivered events -- Stripe sends an event id with every delivery; idempotent processing keyed on that id would be a further hardening step, not implemented here.
• Handle clock skew between Stripe and the database for tie-breaking when two events for the same subscription id arrive out of order -- rare in practice, since Stripe retries are for delivery failures rather than reordering within one subscription's own event stream, but a known gap worth flagging rather than quietly ignoring.

Why This Shipped in the First Place

The original code was reviewed, typechecked, and built cleanly. There was nothing syntactically wrong with it. The four manually-tested happy-path flows -- checkout, cancel, code review, build -- never exercised more than one event in isolation, because that is how manual testing naturally works: one action, one expected result, move on. The bug is purely about the interaction between two events delivered out of order, and that is exactly the category of bug that does not show up until enough real customers have gone through cancel-then-resubscribe in production. No amount of single-event testing, however thorough, would have caught it -- only a test that modeled the actual sequence did.

The lesson underneath the lesson: a fix that passes the test for the bug you found is not automatically correct. It is correct once you have also written the test for the case your fix could plausibly break next -- which, here, was the very next thing checked, and it failed on the first attempt.