Google's Flexible VMs: The Secret Weapon to Keep Your Spark Jobs Alive During AI-Driven Compute Shortages

Google Introduces Flexible VMs for Apache Spark

Google has unveiled flexible virtual machines in its Managed Service for Apache Spark to tackle cluster provisioning failures during compute shortages. As AI infrastructure demand skyrockets, Spark users often face delays when their preferred machine family is unavailable. Flexible VMs let you define an ordered list of acceptable machine families for master, primary worker, and secondary worker nodes, ensuring your jobs keep running.

How Flexible VMs Work

Instead of tying a cluster to a single instance type, flexible VMs allow a mix of machine types and generations in one configuration. This includes Gen2 families like N2 and N2D alongside Gen4 families such as N4 and C4. Storage choices also adapt to the selected machine family at provisioning.

A key feature is the ranking system that determines the order in which machine families are tried. Google recommends specifying at least two machine families in the top-priority rank to improve the chances of obtaining capacity during high demand.

Example Configuration for Production Pipelines

For pipelines built around n2d-standard-16 shapes, Google suggests a tiered approach:

• Rank 0: n2d-standard-16 and n2-standard-16
• Rank 1: n4-standard-16 and n4d-standard-16
• Rank 2: c4-standard-16 and c3-standard-22
• Rank 3: e2-standard-16

For legacy n1-standard-16 workloads, the ranking places n1-standard-16 and n2-standard-16 first, followed by n2d-standard-16, then n4-standard-16 and n4d-standard-16, and finally e2-standard-16. This provides a path to newer architectures while maintaining operational continuity.

Storage Considerations

Google emphasizes Hyperdisk Balanced for newer instance families like N4 and C4, which rely on Hyperdisk for predictable performance. Default IOPS and throughput settings are likely sufficient for many distributed Spark jobs. However, if you want broader fallback options, you may need to accept a change in disk type when workloads move between older and newer machine families.

Operational Trade-offs

Broadening machine options introduces several considerations:

• Quotas: You need enough compute and disk quota for every machine type and storage option listed in your flexible VM rankings, including Hyperdisk.
• Committed spending: Traditional committed use discounts are tied to specific machine families. For savings across multiple VM families and regions, consider compute flexible committed use discounts.
• Performance predictability: Results can vary between machine generations and storage options (e.g., local SSD vs. Hyperdisk). Test your Spark jobs to assess the impact on service levels.

Broader Resilience Steps

Alongside flexible VMs, Google recommends several operational measures to improve job success under constrained capacity:

• Automatic zone selection to place jobs where resources are available.
• Favor smaller machine shapes over heavily contested large-core instances.
• Use autoscaling with sensible upper limits for variable workloads.
• Partial cluster creation to start with a minimum number of primary workers and add more later.
• Regional fallback planning, especially for high-demand areas like us-central1.

The Bigger Picture

As AI workloads compete with conventional analytics for the same infrastructure, Spark users must think beyond a single instance type. The question now is: how many acceptable substitutes can be built into the configuration before a shortage leads to failure? Flexible VMs aim to keep Spark pipelines running when regional or zonal stockouts affect a preferred machine family.