How to define metrics in config, not code

TL;DR

• A metric is data, not a program. Describe it once, and let one engine run it.
• One definition per metric makes duplicates easy to see.
• Do not switch engines because the tests pass. Run the old and the new engine side by side. Switch only when the answers match, or each difference has a written reason.

The problem with metrics as code

In our platform, each business metric was a small function. A new metric meant new code, a code review and a release. Over time, there were a great many of these functions.

Many of them did almost the same thing. Two teams needed the same number, so each team wrote its own version. The versions were slightly different. When the numbers did not agree, nobody could say which one was right.

The code also hid the meaning. To learn what a metric counted, you had to read the function. Business users could not do that, so they asked an engineer.

What a metric definition holds

A metric needs only a few facts:

• The source: the table that holds the data.
• The measure: what to count or add up.
• The filters: which rows to keep.
• The breakdowns: how to split the result, for example by region or by month.

We put these facts in a metric definition. Here is a made-up example:

metric: orders_this_month
source: orders
measure: count
filters:
  - order_date: this_month
breakdowns: [region]

Anyone can read this. A reviewer checks a definition, not a program. Two definitions that are the same are easy to find.

def orders_this_month():
  rows = read('orders')
  rows = [r for r in rows
          if this_month(r)]
  return count(rows, 'region')

metric: orders_this_month
source: orders
measure: count
filters:
  - order_date: this_month
breakdowns: [region]

One engine for every team

I designed and built the engine that reads these definitions and runs them. It is a shared library. Any team can call it with its own definition. Read how the metric engine works for the full design.

Each new version of the engine goes through three stages: development, test and production. A version reaches real users only after it passes in test. So a team that builds on the engine does not get a surprise when the engine changes.

To help people use it, I added two tools:

• A search page. You find any metric, run it without code, and compare its answer with the live system.
• A guide to create a metric. You fill in the definition on one page and preview the result before you save it.

Numbers that depend on who is asking

Some numbers change with the person who asks. A sales manager wants the totals for their own region, split by each person in the team.

The old way built this answer from many small queries, one for each person. The queries were slow, and large regions often timed out.

We changed the approach. The engine now works out the number from a rule, at the moment someone asks. It sends one query in place of many. In our tests, the answers were the same and the result came back much faster.

The data model decides once what a person may see, so no metric needs its own rule. I designed that model. Read how the data protection model works. Then read how to run a security review of a data platform for how we proved it.

Switching engines safely

The new engine must give the same numbers as the old engine, which ran each metric as its own function. Passing tests do not prove that. So we built a side-by-side check.

The check takes a fixed set of sample requests, copied from real use. It sends each request to the old engine and to the new engine. Then it compares the two answers.

The first runs found many differences. We looked at each one. We fixed most of them in the new engine. For the rest, we wrote down the reason. Some were bugs in the old engine, and some needed a decision from the business.

The rule is simple. A release goes live only when the answers match, or when each difference has a written reason.

What I’d do differently

Build the side-by-side check first, before the new engine.

When we started the check, many metrics had no sample request. Without a sample request, you cannot compare the two engines. We found this late, and it slowed the move. Now I would collect a sample request for every metric on day one.