How to Measure the Impact of a Feature Flag

Emit the flag variation alongside the metrics you care about so you can compare outcomes per variation.

To know whether a feature helped, tag events with the flag variation the user received. An analytics pipeline then compares error rate, conversion, or latency between the on and off groups.

Steps

Emit an exposure event when the flag is evaluated.
Tag downstream metrics with the same variation.
Compare metrics across variations before widening the rollout.

Tag metrics with the variation

Terminal

const on = await client.getBooleanValue('new-checkout', false, { targetingKey: user.id });
track('exposure', { flag: 'new-checkout', variation: on ? 'on' : 'off', userId: user.id });
track('checkout_complete', { flag: 'new-checkout', variation: on ? 'on' : 'off' });

Guard promotion on metrics

.github/workflows/ci.yml

jobs:
  gate-rollout:
    runs-on: ubuntu-latest
    steps:
      - run: node scripts/check-flag-metrics.js new-checkout   # exit 1 if variant worse

Gotchas

Without an exposure event you cannot tell who actually saw the feature.
Compare like groups; a rollout skewed toward power users biases the result.