Data work is where cloud costs most often surprise people, because the expensive operations – scanning large volumes, moving data between regions, keeping every intermediate result – do not feel expensive while you are doing them.
Structure storage so queries can skip data
- Partition by the column you filter on most, usually a date.
- Use columnar formats so queries read only the columns they need.
- Keep file sizes moderate; thousands of tiny files are slow and costly.
- Compress, and check the format supports splitting for parallel reads.
- Keep raw, cleaned and aggregated layers separate.
A well-partitioned table can turn a full scan into reading a few percent of the data, which is usually the difference between a cheap query and an expensive one.
The layered model
-
Raw
Immutable, exactly as received. Never edited; the source of truth for reprocessing.
-
Cleaned
Validated, typed, deduplicated. What most queries should read.
-
Aggregated
Pre-computed summaries for dashboards and repeated questions.
Keeping raw data immutable means a bug in transformation is recoverable by reprocessing rather than by apology.
Cost control specific to analytics
| Driver | Control |
|---|---|
| Bytes scanned per query | Partitioning, columnar formats, selective columns. |
| Repeated identical queries | Materialise results or enable result caching. |
| Idle clusters | Auto-terminate; prefer serverless for spiky work. |
| Cross-region reads | Keep compute in the data’s region. |
| Infinite retention | Lifecycle rules on intermediate outputs. |
Isolate the pipeline account
Pipeline workloads run large jobs on schedules and are frequently developed by a different group from the application team. A separate account keeps that spend attributable, keeps large jobs from competing with production, and makes the data access boundary explicit.
Summary
Partition and format so queries scan less, keep raw data immutable, terminate idle compute, and give the pipeline its own account. Analytics costs are mostly a function of how much data you read.
