Kishore Kanthan

Cloud cost

Mapping the cloud bill to the pipelines and teams behind it. Then cutting compute and storage with lighter engines, less indexing and retention rules.

One large cloud bill, mapped to the teams that drive it. Each team's usage then drops and stays down.

The problem

The cloud bill came as one large number. Nobody could say which pipeline or app drove it, so nobody could act on it. Platform usage and the cloud bill also did not move together, which made the bill hard to explain.

Constraints

  • The platform and the cloud bill use different units, and there is no shared key to join them.
  • Cuts must not slow pipelines or break dashboards.
  • Some data must be kept longer than the standard rule allows.

How the work runs

The work runs as a loop. Each fix starts from a mapped cost and ends with a weekly check.

What was built

Cost mapping. The cloud bill is mapped to the platform’s own usage. So each cost lands on the pipeline, app or team that drives it. Savings start from the biggest real cost, not a guess.

A share for each business area. A cost model splits the bill across the business areas. Each area pays a share under an agreed policy, with a yearly price list. The billing data and the usage data have no shared key, so the split goes by category. Read allocating cloud cost without a join key for the method.

A forecast that starts from the right point. A three-year forecast supports the budget. Cost was highest early in the year and then fell. A plan based on the year-on-year change would ask for too much money.

Lighter engines for small jobs. A cluster engine costs a lot to start, even for a small job. Small and medium jobs moved from Spark to a single-machine engine, Polars. Large jobs stay on Spark.

Each job runs on the smallest engine that fits it. Most jobs are small.

Incremental runs, on the roadmap. Some heavy pipelines rebuild their whole output on every run. An incremental run processes only the data that changed since the last run. That changes how each pipeline reads and writes its data. So the move is on the roadmap, one pipeline at a time.

A full rebuild processes every day of data on every run. An incremental run processes only the new day.

No runs that nobody needs. Scheduled pipelines that no one used were paused or deleted. Schedules left running on work branches were removed, so a branch no longer rebuilds data on its own. Old data types that no app reads were retired. Data that no pipeline owns was removed. Restricted views that no app used were removed from the test environment.

Start on an event, not a schedule. One slow export asked for a reserved machine. A measurement showed that most of each run was a wait in the queue. A bigger machine would shorten only the work, not the wait. So the export now starts from an event, and no reserved machine was needed.

Index only what is searched. Business objects are indexed so that apps can search them. By default, every column was indexed. An analysis found which columns apps filter or search on. The rest were taken out of the index, which cut indexing compute and the storage behind the index. The access tables also indexed three spellings of every user ID. Now they index one, as described in Hierarchical access. The benchmark tables had the same problem. They used more indexing than the main tables they summarise. The same lookup was the cause, and the same fix applied.

Retention rules for the whole platform. Every pipeline run leaves a snapshot of its files, and snapshots were kept forever. Deleted data stayed in the trash, and the trash still used storage. Now a set of retention rules applies across the platform. Old snapshots are removed and the trash is emptied. Data that must be kept longer has a stated exception.

What each retention rule keeps. Work branches are short-lived, so their snapshots go after a short time.

Keeping it down

A drop in usage that nobody watches does not last. A weekly report compares platform usage with the cloud bill. It started as a deck made by hand each week. Now an agent skill builds it from one data file.

The report found a surprise. Usage went down, but the bill did not. The cause was a correction in the bill itself. Without the weekly view, the team would have looked for a problem that did not exist.

The next controls are on the roadmap:

  • One cost page that shows the use of each project.
  • A use limit for each project.
  • Alerts when use grows fast, or when data goes unread.
  • A monthly check that each saving still holds.
  • A cost estimate when new work starts, with an agreed owner who pays for it.

Key decisions

  • Map before you cut. When each cost has an owner, the biggest savings are easy to find.
  • Fix the shape of the work, not the size of the machine. The biggest wins came from removing repeat work and right-sizing the engine, not from buying different capacity.
  • Platform-wide rules, with named exceptions. One default for the whole platform, with each exception stated, is easier to audit than a rule per pipeline.
  • Clean up now, prevent with policy as code. Clean-ups remove waste that has built up. Policy as code is on the roadmap. With it, retention and schedule rules are written once, centrally, and every pipeline follows them.
  • Measure like for like. Compare the same weekdays before and after, so a quiet weekend does not look like a saving.

What changed

Platform usage went down and stayed down. Batch compute, index storage and file storage all fell.

The cloud bill also fell in the second quarter of 2026. Other initiatives drove that drop, not this work.

Each change was measured on its own, week by week. The chart shows one of them: storing each user once in the access tables.

Weekly use before and after storing each user once. Indexing compute fell most, then index storage, then file storage. Made-up heights, not to scale.

What to do differently next time

Set up the like-for-like measurement first. Early before-and-after numbers mixed weekdays and weekends, and had to be measured again.