Lakeflow Release notes from Databricks

AI Function tasks for jobs (Beta)

Databricks jobs now support AI Function tasks for parsing, extracting, or classifying files and table rows without writing SQL, currently in beta.

BetaNov 4

Trigger Lakeflow Jobs when upstream jobs complete (Beta)

You can now run a Lakeflow Job when one or more upstream jobs complete, including across workspaces attached to the same Unity Catalog metastore. Configure dependencies on the downstream job. See Trigger jobs when upstream jobs complete .

BetaNov 3

5 releases →

Zerobus Ingest into default storage is now generally available

Ingesting into tables backed by default storage with Zerobus Ingest in Lakeflow Connect is now generally available. See Create or identify the target table .

GANov 2

Monitor data freshness in Lakeflow Connect database connector pipelines (Public Preview)

You can now see how current each destination table is in Lakeflow Connect database connector pipelines, in Public Preview . The new Data freshness column in the pipeline's tables list shows the time since the latest source commit that the pipeline ingested for each change data capture (CDC) table.

PreviewOct 30

Real-time mode in Lakeflow pipelines is now generally available

Real-time mode in Lakeflow pipelines is generally available on classic and serverless compute, providing operational streaming processing latencies as low as five milliseconds.

GAOct 29

Serverless log delivery to Unity Catalog volumes (Beta)

Workspace admins can now deliver serverless job application logs to a Unity Catalog volume for retention and analysis. See Deliver serverless logs to Unity Catalog volumes .

BetaOct 28

Lakeflow Designer October release

The Lakeflow Designer October release adds new AI Function and Prepare capabilities, an HTML operator, shareable full-screen previews, and tools for organizing large flows.

BetaOct 27

Improved Lakeflow Performance Observability is now generally available

Lakeflow Jobs now provide job-level query performance metrics, insights, and optimization suggestions via Genie Code.

GAOct 26

PayPal connector is available in Beta

Databricks Lakeflow Connect now offers a managed PayPal connector in Beta for incremental ingestion of payment data.

BetaOct 24

Google Analytics Aggregated connector is available in Beta

The managed Google Analytics Aggregated connector is now available in beta for Lakeflow Connect, supporting incremental ingestion of GA4 report data.

BetaOct 23

Serverless job environments from pyproject.toml (Beta)

Serverless notebook and Python script tasks can now use pyproject.toml files to configure job environments.

PreviewOct 22

Lakeflow pipelines system tables are now generally available (GA)

The lakeflow system tables pipelines and pipeline_update_timeline are now generally available. These tables record the pipelines and pipeline updates in your account. For a reference of these tables, see Jobs system table reference .

GAOct 21

Asana connector is available in Beta

The managed Asana connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. Use the connector to ingest work management data such as workspaces, teams, projects, tasks, portfolios, and goals from Asana into Databricks. The connector uses OAuth user-to-machine (U2M) authentication.

BetaOct 20

Asynchronous processing with transformWithState is now generally available

Asynchronous processing for the Python row-based transformWithState API is now generally available. Run state operations and user logic concurrently for higher throughput with only minor code changes. See Asynchronous processing with transformWithState .

GAOct 19

MySQL integrated CDC pipeline is now generally available

The integrated CDC pipeline for MySQL is now generally available. An integrated CDC pipeline ingests change data from MySQL into Databricks using a single pipeline, combining extraction and application into one pipeline update. Unlike the standard gateway-based architecture, an integrated CDC pipeline requires no separate ingestion gateway.

GAOct 15

Cribl connector is available in Beta

The managed Cribl connector is now available in Beta in Databricks Lakeflow Connect. Use the connector to ingest Cribl.Cloud Worker Group configuration, worker and health status, organization metadata, and Leader audit and access logs into Databricks.

BetaOct 13

Parameters for Lakeflow pipelines are now generally available

Pipeline parameters are now generally available. Parameterize Lakeflow pipeline SQL source code with key-value pairs so a single pipeline can run against different catalogs, schemas, or tables without editing the source.

GAOct 12

Google Workspace connector is available in Beta

The managed Google Workspace connector is now available in Beta in Databricks Lakeflow Connect. Use the connector to ingest audit activity from Google Workspace applications and services into Databricks. The connector uses OAuth user authorization and supports incremental ingestion.

BetaOct 9

Productboard connector is available in Beta

The managed Productboard connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. Use the connector to ingest product hierarchy, roadmap, objectives, customer feedback notes, and workspace member data from Productboard into Databricks.

BetaOct 8

Use multiple triggers for Lakeflow Jobs (Beta)

Lakeflow Jobs can now use up to five triggers. You can combine trigger types, add multiple triggers of the same type, and pause each trigger independently. See Add multiple triggers to a job .

BetaOct 5

GitLab connector is available in Beta

The managed GitLab connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. Use the connector to ingest repository, project, group, CI/CD, and security data from GitLab into Databricks.

BetaOct 1

Table update triggers on OpenSharing objects and system tables are now generally available

Lakeflow Jobs table update triggers can monitor OpenSharing objects and system tables. See Add a trigger to OpenSharing and system tables .

GASep 30

1Password Event Logs connector (Beta)

The 1Password Event Logs connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector uses the 1Password Events API to ingest audit events, item usages, and sign-in attempts into Databricks. Authentication uses an Events Reporting bearer token.

BetaSep 28

6 releases →

Deleting a Unity Catalog pipeline now retains tables by default

When you delete a Unity Catalog pipeline that uses the default publishing mode, the UI and the Pipelines DELETE API retain associated materialized views, streaming tables, and views by default. The retained tables can be queried but cannot be updated until moved into an active pipeline.

UpdateSep 25

Shopify connector (Beta)

The Shopify connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector ingests e-commerce data such as products, orders, customers, inventory, fulfillments, and discounts from Shopify into Databricks. Authentication uses the Shopify OAuth client credentials grant.

BetaSep 22

JAR tasks on serverless compute will soon be available for workspaces with the compliance security profile enabled

In early October 2026, running JAR tasks on serverless compute will be automatically enabled for workspaces with the compliance security profile enabled. Package your Java or Scala Spark code in a JAR and run it as a JAR task on serverless compute in a Lakeflow Job, without provisioning or managing clusters. See Create and run JARs on serverless compute .

UpdateSep 14

Session restore for serverless jobs is in Public Preview

Restore the Python variables and Spark session from a serverless compute job run into a new interactive notebook, so you can debug a failure or explore results without rerunning the entire job. Databricks captures the snapshot automatically for eligible runs. See Session restore for serverless jobs .

PreviewSep 11

Managed tables in Lakeflow pipelines (Beta)

You can now create managed tables in Lakeflow pipelines and write to them with one or more append flows. This feature is in Beta . You can also use import_checkpoint on an append flow to migrate an existing Structured Streaming workload into a pipeline without reprocessing the source. See create_table , CREATE TABLE ... FLOW (pipelines) , and append_flow .

BetaSep 10

Anaplan connector (Beta)

The Anaplan connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector ingests audit trail events and user account records from your Anaplan tenant into Databricks. Authentication uses an Anaplan CA certificate and its paired private key. See Anaplan connector .

BetaSep 9

Incremental ingestion for Salesforce formula fields in Lakeflow Connect will soon be generally available

Incremental ingestion for Salesforce formula fields in Lakeflow Connect will soon be generally available. Today, Lakeflow Connect ingests formula fields by taking a full snapshot on each pipeline run by default. When incremental formula field ingestion is generally available, full snapshots will no longer be the default.

GASep 8

Zerobus Ingest into default storage is now in Public Preview

Ingesting into tables backed by default storage with Zerobus Ingest in Lakeflow Connect is now in Public Preview . See Create or identify the target table .

PreviewSep 7

Apache Arrow support for Zerobus Ingest is now generally available

Apache Arrow support for Zerobus Ingest in Lakeflow Connect is now generally available. See Use Arrow Flight with Zerobus Ingest .

GASep 1

Pipeline events system table is in Beta

The pipeline_events system table is in Beta . This table records Lakeflow pipelines event log entries for pipelines in your account, capturing lifecycle transitions, flow progress, data quality metrics, errors, and other operational data across all pipelines and workspaces in a region.

BetaAug 31

Atlassian Audit Logs connector (Beta)

The Atlassian Audit Logs connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector ingests organization audit log events, such as user, security, and configuration changes, from Atlassian into Databricks. Authentication uses an Atlassian organization API key.

BetaAug 27

Zerobus Ingest into streaming tables is now generally available

Ingesting into streaming tables with Zerobus Ingest in Lakeflow Connect is now generally available. Writing to a streaming table works the same way as writing to a managed Delta table, with the same limits and quotas. See Create or identify the target table .

GAAug 26

Anysphere Audit Logs connector (Beta)

The Anysphere Audit Logs connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector uses the Anysphere (Cursor) Admin API to incrementally ingest team audit-log events into Databricks. Authentication uses a Cursor Admin API key. See Anysphere Audit Logs connector .

BetaAug 25

Verkada connector (Beta)

The Verkada connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector ingests organization audit logs and access users from your Verkada organization into Databricks. Authentication uses a Verkada organization API key. See Verkada connector .

BetaAug 24

Default Python package repositories are GA for Lakeflow pipelines and classic compute

Configuring default Python package repositories is now generally available for Lakeflow pipelines and for classic compute created via the UI and API. Workspace admins can set private or authenticated registries as the workspace default for these workloads. See Configure default package repositories .

GAAug 21

Salesforce connector mTLS authentication is now generally available

Mutual TLS (mTLS) authentication for the Salesforce connector in Lakeflow Connect is now generally available. With mTLS, Databricks and Salesforce exchange and verify client certificates to establish two-way trust, combined with the OAuth client credentials flow. Choose it for API-only users or when your organization requires mTLS.

GAAug 20

Session restore for serverless jobs is now in Beta

Restore the Python variables and Spark session from a serverless compute job run into a new interactive notebook, so you can debug a failure or explore results without rerunning the job. Databricks automatically captures a state snapshot when an eligible run fails, runs 30 minutes or longer, or is cancelled, and keeps it for 7 days after the run completes.

BetaAug 17

Workiva connector (Beta)

The Workiva connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector ingests audit activity, users, and roles from your Workiva organization into Databricks. Authentication uses Workiva OAuth2 client credentials. See Workiva connector .

BetaAug 14

Gmail connector in Lakeflow Connect (Beta)

You can now ingest data from a Gmail mailbox into Databricks using the managed Gmail connector in Lakeflow Connect. The connector ingests messages, labels, drafts, filters, and profile data using a Google service account, and syncs the messages and message_labels tables incrementally through the Gmail History API. This feature is in Beta .

BetaAug 13

Amplitude connector (Beta)

The Amplitude connector in Lakeflow Connect is now in Beta. Ingest product analytics event data along with event, cohort, and annotation metadata from Amplitude. See Amplitude connector .

BetaAug 12

REPLACE USING flows for standalone streaming tables (Beta)

REPLACE USING flows are now available for standalone streaming tables in Beta . Use a REPLACE USING flow when your source is a series of partial snapshots keyed by column. On each update, the flow replaces all rows that match the specified key columns and leaves all other rows unchanged.

BetaAug 11

Notion connector (Beta)

The managed Notion connector in Lakeflow Connect is now available in Beta. The connector ingests pages, data sources, databases, blocks, comments, and users from Notion into Databricks. See Notion connector .

BetaAug 10

Change data feed on materialized views (Beta)

You can now read a change data feed from a materialized view created in a Lakeflow pipeline or in Databricks SQL, using Databricks Runtime 18 LTS or above. Use this to replicate materialized view changes to destinations outside Databricks, or to keep a history of materialized view changes for auditing and reporting.

BetaAug 7

JAR tasks on serverless compute are now generally available

Package Scala or Java code as a JAR and run it as a task in a Lakeflow Job on serverless compute, without provisioning or managing clusters. See Create and run JARs on serverless compute .

GAAug 6

REPLACE USING flows for pipelines (Beta)

REPLACE USING flows are now available in Lakeflow pipelines in Beta . Use a REPLACE USING flow when your source is a series of partial snapshots keyed by column. On each update, the flow replaces all rows that match the specified key columns and leaves all other rows unchanged.

BetaAug 5

Google Drive connector (GA)

Both Google Drive connectors in Lakeflow Connect are now generally available: the managed connector for fully managed ingestion pipelines, and the standard connector for custom pipelines built with Spark and SQL APIs.

GAAug 4

4 releases →

Environment variables for serverless jobs are in Beta

Environment variables for serverless Lakeflow Jobs tasks are in Beta . You can define named environment variable entries at the job level and select one for each task to pass deployment-specific configuration to your application code. A workspace admin must enable the preview from the workspace Previews page.

BetaAug 3

OpenAI connector (Beta)

The OpenAI connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector uses the OpenAI Organization Admin API to ingest organization administration data, such as users, projects, API keys, usage, costs, and audit logs, from OpenAI into Databricks.

BetaJul 31

Unit testing for pipelines (Beta)

You can now write and run Python unit tests for your Spark Declarative Pipelines on Lakeflow transformation logic directly in the Lakeflow Pipelines Editor. Validate Python and SQL transformations against mock data using catalog-table redirection, including Auto CDC, streaming tables, expectations, and append flows.

BetaJul 26

Lakeflow Designer Summer Release

The Lakeflow Designer Summer Release adds new operators, expands the Output and Join operators, and improves the canvas editing experience. The following updates are now available: Prepare operator : Use the new Prepare operator for common data-cleansing tasks, including changing text case and data types, replacing values, and applying formulas.

UpdateJul 24

C++ SDK for the Zerobus Ingest connector is now available in Beta

A C++ SDK for the Zerobus Ingest connector in Lakeflow Connect is now available in Beta. The SDK provides a high-throughput gRPC client for ingesting records directly into Unity Catalog Delta tables, with support for JSON and Protocol Buffers. See Use Zerobus Ingest .

BetaJul 23

Lakeflow Designer support for compliance security profile standards

Lakeflow Designer now supports all compliance security profile standards supported by Databricks and is available by default for workspaces with the compliance security profile enabled. For a full list of supported compliance standards, see Compliance security profile . See Lakeflow Designer .

UpdateJul 22

Reddit Ads connector is now available in Beta

The managed Reddit Ads connector in Lakeflow Connect is now available in Beta. You can use the connector to ingest advertising campaign data and performance metrics from Reddit Ads into Databricks. See Reddit Ads connector .

BetaJul 21

Anthropic managed ingestion connector (Beta)

The Anthropic connector in Lakeflow Connect is now available in Beta. The connector allows you to ingest Compliance API data , audit activity, organizations, users, roles, groups, settings, and Claude chats and messages , from Anthropic into Databricks. See Anthropic connector .

BetaJul 17

Query-based connectors in Lakeflow Connect will soon be available by default for workspaces with the compliance security profile enabled

Query-based connectors in Lakeflow Connect will soon be available by default for workspaces with the compliance security profile enabled. Query-based connectors ingest data from databases by querying the source directly using a cursor column, without requiring change data capture (CDC) configuration or an ingestion gateway.

UpdateJul 14

Strac connector (Beta)

The Strac connector is now available in Lakeflow Connect in Beta . Workspace admins can control access to this feature from the Previews page. The connector ingests data-security audit events from your Strac instance into Databricks. Authentication uses a Strac API key. See Strac connector .

BetaJul 9