14 releases · 0 this month
Dataflow now supports streaming high availability for Streaming Engine jobs ( Preview ). When you enable the requires_high_availability service option, Dataflow replicates backend state across multiple zones in a region to protect your job from single-zone failures. For more information, see Streaming high availability .
Dataflow Auto VM Selection now supports 4th generation Compute Engine machine series. When Auto VM Selection is enabled, the service prioritizes these newer generation VMs for improved performance.
You can now pause a Dataflow batch job using the pause_on_failure service option. This feature lets you preserve the state of your batch pipeline job, address external issues, and resume processing without losing completed work. You can use this option to automatically pause a job on failure, or use this feature to manually pause a job when you chose to.
You can now use Hyperdisk Balanced disks for Dataflow worker VMs. With Hyperdisk Balanced disks, you can provision IOPS and throughput independently of disk size by using the diskProvisionedIOPS and diskProvisionedThroughput pipeline options (Java SDK) or disk_provisioned_iops and disk_provisioned_throughput_mibps pipeline options (Python and Go SDKs).
Dataflow now supports NVIDIA RTX Pro 6000 GPUs. You can use this GPU model to run your Apache Beam pipelines on Dataflow. RTX Pro 6000 GPUs are recommended for large, medium, and small model inference workloads. To configure your workers with this GPU model, set the accelerator type to nvidia-rtx-pro-6000 .
Dataflow has updated and expanded its pipeline update features for streaming jobs: Automated stop-and-replace updates : You can perform automated, declarative stop-and-replace updates to streaming jobs. Parallel updates with the same job name : When you perform automated parallel updates, you can use the same job name for the new replacement job.
Dataflow job builder now supports external Iceberg REST Catalogs as a source. You can now ingest data from external Apache Iceberg REST catalogs (IRC) directly into Lakehouse tables using Dataflow's job builder UI without writing code. For more information, see Import data from external Iceberg catalogs to Lakehouse using Dataflow .
When you use min_ram or cpu_count resource hints for pipeline steps that don't require accelerators, Auto VM Selection (Instance Flexibility) is enabled automatically. With Auto VM Selection, workers are provisioned from a curated list of machine types that meet your RAM and CPU requirements.
Dataflow support for the C4A machine series of Arm processors is now generally available. Arm-based VMs are optimized for power efficiency and can provide improved price-performance for many workloads. For more information, see Use Arm VMs on Dataflow .
Dataflow Managed I/O now supports rolling upgrades for streaming jobs. With this feature, Dataflow upgrades your Managed I/O connectors in running pipelines as new connector versions become available. For more information, see Automatic upgrades .
Dataflow is available in the Bangkok ( asia-southeast3 ) region. Learn more about Google Cloud locations .
Dataflow now serves a notice for when the Dataflow Runner v2 container image of a streaming pipeline will be upgraded. To use a new image and avoid the scheduled maintenance, launch a replacement job before the upgrade. For more information, see Runner v2 harness update .
Dataflow now supports speculative execution for batch pipelines. This feature mitigates the impact of slow-running tasks ( stragglers ) by launching a redundant execution of these tasks. The first task to finish is used, and the other is canceled, which can improve the overall completion time of your pipeline. This feature is generally available.
For jobs that use GPUs, Dataflow now supports the flex-start provisioning model. This flex-start provisioning model can improve your ability to get access to constrained GPU resources for short-duration workloads. This feature is available in Preview and is for batch pipelines only. For more information, see Configure a provisioning model .
Stream and batch processing on Apache Beam.
Other Google Cloud products: AlloyDB, BigQuery, Cloud Run, Cloud SQL, Dataplex Universal Catalog, Gemini API, Gemini CLI, Gemini Enterprise, Gemini Enterprise Agent Platform, Google Kubernetes Engine, Looker, Pub/Sub, Spanner
| Release-notes page | Site | Feed |
|---|---|---|
| Cloud SQL for MySQL docs.cloud.google.com | docs.cloud.google.com | RSS |
| Cloud SQL for PostgreSQL docs.cloud.google.com | docs.cloud.google.com | RSS |
| Colab Enterprise docs.cloud.google.com | docs.cloud.google.com | None |
| Dataflow docs.cloud.google.com | docs.cloud.google.com | RSS |
| Dataproc docs.cloud.google.com | docs.cloud.google.com | RSS |
| Firestore docs.cloud.google.com | docs.cloud.google.com | None |
| Gemini Enterprise – Business edition release notes support.google.com | support.google.com | None |
| Google Cloud all release notes docs.cloud.google.com | docs.cloud.google.com | RSS |
| Pub/Sub docs.cloud.google.com | docs.cloud.google.com | RSS |
| Vertex AI / Agent Platform docs.cloud.google.com | docs.cloud.google.com | RSS |
| Vertex AI generative AI docs.cloud.google.com | docs.cloud.google.com | None |
| Gemini Apps' release updates & improvements gemini.google | gemini.google | None |
Releasesby Glyph