- Published on
GKE Pod Snapshots Cut Model Load Times, and Move the Work to Snapshot Lifecycle Management
- Authors
- Name
- DeepHorus
Google Cloud reported GKE Pod snapshot benchmarks showing startup-latency cuts as high as 89%. A 70-billion-parameter model loaded in 37 seconds, and an 8-billion-parameter model in 15. The feature reached general availability in May on GKE 1.35.3-gke.1234000 or later, letting replacement Pods resume workloads rather than initialize them again.
Unlike caching, a snapshot preserves CPU and GPU memory, registers, threads, open file descriptors and other process state. It also captures the container root filesystem, EmptyDir volumes and tmpfs mounts. A replica resumes without rerunning the model-loading initialization that dominates large-model startup.
GVisor enables the feature, but Pods must run in GKE Sandbox, where that runtime resides. Autopilot includes it; Standard clusters need a gVisor-enabled node pool. A node agent manages snapshots, a controller removes obsolete copies, and Cloud Storage holds data.
Two custom resources configure the system. PodSnapshotStorageConfig names the bucket; PodSnapshotPolicy selects labeled Pods, sets workload or manual triggers, and defines lastAccessTimeout and a per-group snapshot limit.
Codeway’s Retake platform used a custom artifact cache to reach about one minute. Lead DevOps engineer Ahmet Furkan Çomak said Pod snapshots cut that to eight seconds. The team now launches H100 instances for a specific job and shuts them down on completion.
Full snapshots impose strict compatibility rules. GKE hashes the “distilled Pod spec” and requires matching machine series, CPU architecture, gVisor kernel and GPU driver; if none matches, the Pod starts normally. Rootfs-only snapshots skip the check and can cross machine families, including to E2, but omit process memory.
Practitioners see lifecycle management as harder because upgrades can invalidate snapshots and erase the benefit without an error. Applications must recreate keys, certificates and environment values, and re-establish connections; persistent volumes and network rules remain outside checkpoints. Google’s fallback prioritizes availability, making retention and compatibility planning central to adoption.
