Machine learning work has an awkward shape for cloud billing: bursts of expensive compute separated by long periods of nothing. Handled carelessly, the idle periods cost as much as the useful ones.
Separate the phases
-
Exploration
Notebooks and small samples. Modest compute, high interactivity.
-
Training
Short, intense, expensive. Should start and stop cleanly.
-
Evaluation
Moderate compute, often repeated. Worth scripting.
-
Serving
Continuous, latency-sensitive, sized differently from training.
Treating these as one environment is the most common and most expensive mistake. A training-sized instance left running for interactive work is pure waste.
Checkpoint everything
- Write checkpoints to object storage at regular intervals, not only at the end.
- Make training resumable from the latest checkpoint without manual steps.
- Log metrics externally so a lost instance does not lose the run history.
- Version datasets and code together so a result can be reproduced.
- Shut the instance down automatically when the job finishes.
Resumable training is what makes interruptible capacity usable, and interruptible capacity is where the savings are.
Data movement is often the bottleneck
| Stage | Watch for |
|---|---|
| Getting data in | Initial transfer can dominate a short project. |
| Reading during training | Storage throughput, not accelerator speed, may be the limit. |
| Cross-region access | Charged and slow; keep data and compute together. |
| Egress of results | Leaving the provider is usually the charged direction. |
| Idle accelerators | Provisioned and waiting on a slow data path. |
Quotas bite hardest here
Specialised compute is the most constrained category on every platform. Check the quota in your chosen region before planning a schedule around it, and request increases well ahead of when you need the capacity.
Summary
Split exploration from training from serving, checkpoint so jobs are resumable, keep data next to compute, and shut things down automatically. The accelerators are rarely the thing that wastes money; waiting is.
