Kishore Kanthan
Article · · 3 min read

How to cut compute and storage use on a data platform

Where the money went on a busy data platform, and the fixes that brought compute and storage usage down.

One large cloud bill, mapped to the teams that drive it. Each team's usage then drops and stays down.

TL;DR

  • Show each team its own share of the bill first. Teams that see their own cost ask better questions about it.
  • Fix the shape of the work, not the size of the machine. Removing duplicate work cut usage the most.
  • Compare like with like: the same weekdays before and after. A quiet weekend is not a drop in usage.

Where the money went

Our data platform runs on a public cloud. Most of the bill came from three things:

  • Compute for the data jobs that run every day.
  • Search indexing: the platform indexes data so that apps can find it fast.
  • Storage for the data and for its indexes.

The bill came as one large number. It did not say which team or which job caused the cost. So nobody could act on it.

Allocate before you cut

First, we split the bill across the business areas. Each area paid a share under an agreed policy, with a yearly price list. When teams see their own cost, they ask better questions about it. The cloud cost case study covers the whole effort.

The billing data and the usage data had no shared key. So we split the bill by category: pipeline compute, interactive compute, file storage, index storage and index compute. Then we added the platform overhead on top. Read allocating cloud cost without a join key for how we split it.

We also built a forecast for the next three years. Here, timing mattered. Cost was highest early in the year and then fell. Other initiatives drove that fall, not the fixes below. A plan based on the year-on-year change would ask for too much money.

Search: one lookup in place of three

One finding cut usage the most.

A search index works like the index at the back of a book. It maps each value to the rows that hold it. All values are made up.

The platform stored each user ID in three forms: upper case, lower case and mixed case. Many apps used a lookup that held all three. So the index held every user three times.

Each user ID was stored in three forms. So the index held three entries for each person, and grew to three times the size it needed.

We replaced it with one shared lookup that holds each ID once, in one form. Search indexing now uses far less compute, and the indexes use less storage.

The fix: one shared lookup holds each ID once, in one form. The index needs one entry for each person, so it is smaller and needs less compute to build.

We found a related problem in the benchmark tables. A benchmark widens a person’s view to their whole sales team. These tables used more indexing than the main tables they summarised. We traced this to the same user lookup, and fixed it.

Compute: stop doing work twice

Some jobs process a whole table every day, when only a small part has changed. Moving them to process only the new and changed rows is on the roadmap, one pipeline at a time.

One export was slow. The team wanted a dedicated, reserved machine for it. I measured the run first. Most of the time was waiting in a queue, not working. So a bigger machine would help only a little. We started the export from an event in place of a fixed schedule, and it no longer waited.

Most of the export's time was a wait in the queue. A bigger machine shortens only the work; an event trigger removes the wait. Bar lengths show shares of time, not real durations.

That saved the cost of the reserved machine too.

Storage: keep less, for less time

We cleaned up in three steps:

  1. Switch off jobs that nobody uses. Each one costs compute and storage every day.
  2. Remove old and unused data, for example test copies and data views that no app reads.
  3. Set retention rules. Each kind of data has a time limit. After that time, the platform deletes it.

Keeping it down

A drop in usage that nobody watches does not last. So we track it every week.

A weekly report compares platform usage with the cloud bill. It found a surprise: usage went down, but the bill did not. The cause was a correction in the bill itself. Without the weekly view, we would have looked for a problem that did not exist.

Making cost part of every new request is on the roadmap. The plan: estimate the cost when the work starts, agree who pays for it, and enforce cost limits with rules in code.

Measuring a drop in usage honestly

Be careful with before-and-after numbers.

Our first “after” window ran from Thursday to Sunday. Weekends are quiet, so the drop looked bigger than it was. We had to measure again, on the same weekdays before and after.

Also, keep usage and the bill apart. Usage that you avoid is not the same as a smaller bill. Say which one you mean.

What I’d do differently

Set up the like-for-like measurement first, before the first fix. Then every drop in usage is measured the same way, and nobody has to measure it again.