Compute, Storage & Networking

That last point drives the economics. An 8-GPU H100 node can list around $32/hr per node (illustrative). If your data pipeline starves the GPUs and they sit at 40% utilization, you are paying full price for a 40%-effective machine. Spot/preemptible pricing can cut the hourly rate substantially in exchange for possible interruption, which is why checkpointing (below) matters so much.

Storage: where datasets, checkpoints, and artifacts live

Two storage models cover almost everything in an AI stack, and choosing correctly is mostly about access pattern.

Object storage (Amazon S3, Google Cloud Storage, Azure Blob) is a flat namespace of immutable objects addressed by key and accessed over an HTTP API. It offers roughly eleven nines of durability (99.999999999%), effectively unlimited capacity, very high aggregate throughput, and the lowest cost per GB (around $0.02/GB-month for a standard tier, illustrative). Its weakness is per-request latency (tens of milliseconds) and that it is not a POSIX filesystem, you cannot mmap it or do fast random in-place writes. It is the correct home for datasets, checkpoints, model weights, and logs.

Block storage (Amazon EBS, GCP Persistent Disk, Azure Managed Disks) is a raw virtual disk attached to a single VM and formatted with a filesystem. It gives low latency and high IOPS for random access, at a higher price (around $0.08–0.125/GB-month, illustrative) and a capacity you provision up front. It is the right home for the OS/boot volume, local scratch space, a hot cache of the current dataset shard, and databases.

A common, effective pattern combines them: keep the durable copy of a dataset in object storage, and stream shards onto fast local block storage (often NVMe SSD) as a cache during training.

Networking: regions, zones, VPCs, and the cost of moving bytes

Cloud capacity is organized geographically:

• A region is a geographic area (for example us-east-1, europe-west4). Cross-region distances mean tens to well over a hundred milliseconds of round-trip latency.
• An availability zone (AZ, or zone) is an isolated datacenter within a region. Zones in the same region are linked by fast, low-latency networking; spreading across them buys fault tolerance.
• A VPC (Virtual Private Cloud) is your private virtual network, subnets per zone, routing, and security groups, that isolates your training cluster and connects it privately to storage and other services.

The part that surprises people is data-transfer (egress) pricing. The rough rules across major clouds:

• Ingress (data into the cloud) is generally free.
• Internet egress (data out to the public internet) is charged per GB, around $0.09/GB for the first tier (illustrative), tapering with volume.
• Cross-region transfer between the provider's own regions is charged per GB, around $0.02/GB (illustrative).
• Cross-AZ transfer within a region is typically around $0.01/GB per direction (illustrative).
• Same-region access to object storage from compute (for example S3 to EC2 in the same region) usually incurs no transfer charge, you pay only tiny per-request fees.

Why data locality decides your GPU bill

Colocating data with the GPUs that read it matters on three axes at once:

• Bandwidth. In-region object storage delivers very high aggregate throughput to a cluster, enough to keep many GPUs fed. A cross-region path adds latency and realistically caps sustained throughput, so the pipeline becomes the bottleneck.
• Cost. Reading a dataset in-region is effectively free on transfer; reading it across a region boundary charges per GB every pass. Training reads the full dataset once per epoch, so a cross-region read is not a one-time cost, it recurs on every epoch.
• Latency. Cross-region round trips of 50–150 ms stall data loaders; single-digit-millisecond in-region access does not.

Because GPU nodes are billed by wall-clock time, anything that starves them, a slow, distant, or expensive data path, converts directly into paying full price for idle accelerators. Data locality is therefore not just a transfer-bill concern; it is the primary lever on effective GPU cost.

Worked example: moving a 2 TB dataset across regions

Suppose your dataset lives in an object-storage bucket in Region A, but the GPU capacity you could get is in Region B. Using billing units where $1\ \text{TB} = 1000\ \text{GB}$:

$2\ \text{TB} = 2000\ \text{GB}$

One-time replication of the dataset into Region B, at an illustrative cross-region rate of $0.02/GB:

$2000\ \text{GB} \times \$0.02/\text{GB} = \$40$

That $40 is unremarkable. The trap is streaming the data across the region boundary on every epoch instead of copying once. For a 30-epoch run:

$30 \times \$40 = \$1{,}200$

The same 2 TB, read from in-region object storage into GPU nodes, costs about $0 in transfer (plus negligible per-request fees). So the fix is a single replication step:

$\$40\ \text{(replicate once)} + \approx \$0\ \text{(in-region reads} \times 30) \;\Rightarrow\; \text{saves} \approx \$1{,}160$

Two more numbers worth internalizing:

• Egress to the public internet (for example, pulling the dataset down to on-prem) at an illustrative $0.09/GB: $2000 \times 0.09 = 180$, i.e. $180.
• Cross-AZ, same region, at an illustrative $0.01/GB per direction, even staying inside one region, a chatty pattern that reads across zones every epoch costs $2000 \times 0.01 = 20$, i.e. $20 per pass, or $600 over 30 epochs. In-region object storage sidesteps this because it is not tied to a single zone.

A typical training data flow that avoids the trap

1. 1Land raw data in a region-local bucketIngest into object storage in the same region where you will run GPUs. Ingress is free; this becomes your durable source of truth.
2. 2Preprocess on CPU instancesClean, tokenize, and repack into large sequential shards (WebDataset, TFRecord, Parquet). Write results back to object storage in the same region.
3. 3Provision GPU nodes in that same region and VPCKeep accelerators and data in one region; prefer one zone for tightly-coupled multi-node jobs to minimize cross-AZ traffic and latency.
4. 4Stream shards to the GPUs, cache hot data on local block storageIn-region object reads are effectively free on transfer; NVMe/block scratch absorbs random access and repeated reads across epochs.
5. 5Write checkpoints back to object storage periodicallyDurable, cheap, and zone-independent. On spot/preemptible nodes this lets you resume after an interruption instead of restarting.
6. 6Persist final artifacts, then tear down GPU nodesStopping the instances is what stops the largest charge. Keep the cheap object-storage artifacts; release the expensive compute promptly.

Further reading

• Amazon S3 pricing, object-storage GB-month, request, and data-transfer rates.
• Amazon EC2 On-Demand pricing, CPU and GPU instance hourly rates, including data-transfer notes.
• AWS: Overview of Data Transfer Costs for Common Architectures, a clear breakdown of egress, cross-region, and cross-AZ charges.
• Amazon EBS pricing, block-storage provisioning, IOPS, and throughput costs.
• Google Cloud Storage classes, object-storage tiers and their trade-offs.
• Google Cloud VPC network pricing, egress, inter-region, and inter-zone rates.
• Google Cloud: Regions and zones, how the geographic hierarchy is organized.
• Azure Blob Storage introduction, object-storage concepts on Azure.
• Azure: Availability zones overview, regions versus zones and resilience.