Introducing the Apache Spark Optimization Agent for AWS Glue ETL
AWS Glue ETL introduced the Apache Spark Optimization Agent, which analyzes completed Spark runs to deliver automated cost and performance recommendations.
Also shipped Nov 18, 2026AWS in November 2026 →
Genie Code CLI is now generally available
Databricks has made the Genie Code CLI generally available, aligning its management with workspace Genie Code. Administrators can oversee a single Genie Code budget across both the CLI and workspace, while users can apply free monthly usage in the CLI.
Smart Routing now supports external models for coding agents (Beta)
Databricks added support for routing coding-agent subagents to external model providers in the Unity Gateway Managed Configuration beta.
V3 progress control | CodeRabbit Cloud, V3 review settings preview
CodeRabbit updated progress controls in the V3 review settings preview, allowing progress_updates configuration to control review-progress reporting format.
Hyperdrive, Workers VPC - Custom CA and client certificates (mTLS) for Workers VPC
Cloudflare Workers VPC now supports custom certificate authority (CA) verification and mutual TLS (mTLS) authentication for private VPC Services.
Spark Monitoring UI (General availability)
Snowflake announced the general availability of the Spark Monitoring UI in Snowsight for tracking and troubleshooting Snowpark Connect for Spark jobs and sessions.
Gemini 4 Neon (GA)
Google Cloud announced the general availability of Gemini 4 Neon in Gemini Enterprise Agent Platform, supporting a 1M-token context window and four thinking levels.
More from AWS Glue and Lake FormationEvery AWS Glue and Lake Formation release →
AWS Glue announces automatic version upgrades for Apache Spark jobs
AWS Glue introduced automatic version upgrades for Apache Spark jobs set to version Auto. The feature evaluates job compatibility, obtains consent, and validates upgraded job runs against output profiles before updating runtime versions.
AWS Glue adds 100 new native connectors for popular SaaS applications
AWS Glue now includes 100 native connectors for SaaS and REST API applications, enabling data integration without custom code. These connectors allow teams to reuse connections across ETL jobs with existing permission models.
AWS Glue Data Catalog now supports catalog-vended credentials with Apache Iceberg REST catalog federation
AWS Glue Data Catalog now supports vended credentials for Apache Iceberg REST catalog federation. This allows users to read external Iceberg tables using coarse-grained grants and temporary security credentials scoped to S3 prefixes.
Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg
Amazon S3 Tables now supports managed migration of Apache Hive tables to Apache Iceberg. The service handles data validation, compatibility checks, type conversions, and file recompression as a managed, API-driven operation.
AWS Glue Data Catalog now supports Apache Iceberg table optimization in the US West (N. California) Region
Automatic table optimization for Apache Iceberg is now available in the US West (N. California) region. It compacts small files and removes unreferenced snapshots to improve query performance.
AWS Glue Data Catalog now supports OAuth for Apache Iceberg REST Catalog APIs
AWS Glue Data Catalog now supports OAuth 2.0 bearer token authentication for Apache Iceberg REST Catalog APIs, allowing secure federated access for third-party analytics platforms.
Amazon EMR now supports native data quality checks in Apache Spark jobs
Amazon EMR now validates data quality natively within Apache Spark jobs. Data engineering teams can catch quality issues inline before bad data reaches downstream systems, without building or maintaining separate validation infrastructure.
AWS Glue Data Catalog now supports table optimization, statistics, and crawlers for Apache Iceberg V3
AWS Glue Data Catalog now supports table optimization, statistics, and crawlers for Apache Iceberg Version 3 (V3) tables. With these new capabilities, you can automatically maintain V3 tables, optimize them for query performance, and discover them in Amazon S3.