How to allocate AWS cost for data platforms
Splitting a shared cloud bill across teams when the billing data and the usage data share no key. Then turning usage into an estimate.
TL;DR
- The bill and the usage data share no ID. So join them on the cost category.
- Each team pays its share of each category. Shared services are overhead, split by usage.
- Divide each category’s cost by its usage to get a rate card. Use it to estimate new work.
The problem
The bill lists cost by AWS service. The platform measures usage by team, in its own units. No column appears on both sides.
The bill
| Service | Cost |
|---|---|
| EC2 compute | 1,500 |
| EC2 Other | 300 |
| S3 | 300 |
| RDS | 200 |
| Small lines | 100 |
| Total | 2,400 |
The usage
| Team | Usage |
|---|---|
| Team A | 1,000 pipeline hours |
| Team A | 500 GB of indexes |
| Team B | 3,000 pipeline hours |
| Team B | 6,000 GB of files |
What is on the bill
- EC2 compute: the machines that run jobs, queries and indexes. Usually the largest line.
- EC2 Other: disks (EBS), snapshots, NAT gateways and data transfer. Easy to miss.
- S3: object storage for data files, logs and backups.
- RDS: databases that hold the platform’s own metadata.
- Small lines: CloudTrail, CloudWatch, load balancers and keys.
Map each service to a category
Both sides describe the same kinds of work. So a category links them. Tag each machine by its role, so the cost tools can split EC2.
Pipeline compute
Interactive compute
Index compute
Index storage
File storage
Overhead: platform
Overhead: network
Snapshots follow the disks they copy. An untagged line goes to overhead until someone tags it.
Split the cost
- 1Map the billEach line to one category
- 2Measure the usageEach team's share of each category
- 3Split the costCategory cost times the team's share
- 4Add the overheadBy each team's share of usage cost
Here is one team:
| Category | Bill | Team pays |
|---|---|---|
| Pipeline compute | 800 | 200 (25%) |
| Interactive compute | 400 | 20 (5%) |
| File storage | 300 | 30 (10%) |
| Index storage | 200 | 100 (50%) |
| Index compute | 300 | 150 (50%) |
| Usage | 2,000 | 500 (25%) |
| Overhead | 400 | 100 (25%) |
| Total | 2,400 | 600 |
Platform overhead
CloudTrail, the network and the platform databases belong to no single team. No team can switch them off alone. So each team pays the same share of them as its share of usage.
Keep the overhead on its own line. Hidden inside the categories, it blurs a team’s own usage.
Check that it adds up
The team totals must add up to the bill. If they do not, a bill line has no category, or a category has no usage.
From usage to an estimate
The split says what a team spent. A rate card says what new work will cost.
For each category, divide its cost by the platform’s total usage. Pipeline compute cost 800 for 4,000 hours, so it costs 0.20 an hour. The overhead, 400 on 2,000 of usage, adds 20%.
| Category | Unit | Rate |
|---|---|---|
| Pipeline compute | Hour | 0.20 |
| Interactive compute | Hour | 0.40 |
| File storage | GB-month | 0.03 |
| Index storage | GB-month | 0.20 |
| Index compute | Hour | 0.20 |
| Overhead | Usage cost | +20% |
A team that plans a new pipeline states its usage. The rate card turns it into a cost:
| Item | Usage | Cost |
|---|---|---|
| Pipeline compute | 300 hours | 60 |
| File storage | 2,000 GB | 60 |
| Index storage | 100 GB | 20 |
| Index compute | 50 hours | 10 |
| Overhead | 20% of 150 | 30 |
| Estimate | A month | 180 |
A smaller index or fewer runs changes one row, and the effect is clear.
We refreshed the rate card once a year, from a full year of the bill. A forecast is then the forecast usage times the rate. The monthly split still uses the real bill.
Why it helps
Each team sees its cost by category, so it knows what to fix. Here, half of all index cost is one team’s. So the first question is which of its columns need an index.
The rate card moves the cost talk earlier: before the build, not after the bill.
This is the method behind the cloud cost work.