Data Bricks

Databricks and OpenMetadata for a partner data fabric

Databricks provides the platform to integrate, govern, share, analyze, and visualize partner data products. OpenMetadata provides a searchable catalog where people across departments can find those products, understand what they mean, identify who owns them, and follow the approved path to access them. In a federated model, each partner can retain ownership of its source systems and publish approved products for others to use. The fabric connects those products through shared access mechanisms, business definitions, and lineage.

How Databricks connects partner data products

A data product is a maintained set of assets published for a defined use. For example, a logistics partner could publish a Shipment Performance product with a curated table, documented delivery metrics, a daily refresh commitment, and a named owner. A customer operations team could combine that product with order data from another partner to report delivery performance and forecast delays.

Capability	How it works across partner organizations
Data integration	Lakeflow Connect ingests data from supported databases, SaaS applications, files, and streaming sources. Lakeflow pipelines transform and validate the data into reusable tables and views. Where a supported source can be queried in place, Lakehouse Federation provides another integration path.
Orchestration	Lakeflow Jobs coordinates ingestion, quality checks, transformations, and downstream workloads. A job can check that both partners’ daily feeds have arrived and passed validation before rebuilding a combined dataset or running a forecast.
Governance and sharing	Unity Catalog manages privileges and auditing for Databricks assets. Partners publish approved assets through OpenSharing, which supports the Delta Sharing protocol. A recipient with a Unity Catalog-enabled workspace can access a share in Databricks. Other recipients can use supported clients through the open sharing option. The share provides read-only access to the published data.
Advanced analytics	Analysts can join approved partner datasets with Databricks SQL or use Python and Spark for deeper analysis. Data scientists can build forecasts, detect anomalies, or train models on the combined data and use MLflow to track experiments, evaluate models, and manage deployment.
Visualization	Teams can build Databricks AI/BI dashboards or connect supported external BI tools such as Power BI. A shared data product and a published dashboard have separate access settings. Databricks dashboards can run queries using either the publisher’s permissions or each viewer’s permissions, depending on the publishing configuration.

Sharing, federation, and ingestion serve different operational needs. Use a share when a partner already maintains a product and can grant direct access to it. Use federation to query a supported source in place. Use ingestion when the receiving organization needs to retain its own history, standardize incoming records, or process a local copy on its own schedule. Databricks on AWS

How OpenMetadata works with Databricks

OpenMetadata’s Unity Catalog connector runs scheduled ingestion workflows against authorized Databricks environments. It can catalog technical metadata and tags, import usage statistics from query history, and extract table- and column-level lineage from Databricks system tables. Each workflow depends on the connector’s configuration and permissions. Connecting other systems, such as source databases and BI platforms, lets OpenMetadata show assets beyond Databricks in the same catalog.

What users need to know	What OpenMetadata provides
What data is available?	Users can search for a business term such as on-time delivery and browse related products and assets by business domain. A product entry can group its published tables, dashboards, and other assets.
What does it mean, and can I use it?	Asset descriptions, schemas, column definitions, glossary terms, classifications, and data contracts explain the product’s purpose and limitations. Where configured, profiling and quality results help users assess whether the data fits their use case.
Who owns it, and how do I get access?	The product entry can identify the accountable team and document the access route, such as a Databricks catalog and table, an approved partner share, or a dashboard. The entry can also point users to the appropriate approval process. Unity Catalog, the share, or the downstream platform enforces the actual permissions.
Where did it come from, and what depends on it?	Upstream lineage shows the sources and transformations that produced an asset. Downstream lineage shows the datasets, reports, or models that depend on it. Where column-level lineage is available, users can trace an individual field into a downstream calculation and assess the impact of a proposed change.

End-to-end lineage requires coverage on both sides of a partner boundary. For example, an analyst could trace an On-Time Delivery dashboard back to a combined analytics table, then to the shared shipment product and the partner’s source table. That trace depends on collecting lineage from the publisher, the recipient, and the BI platform, and linking the publisher’s product to the recipient’s representation of it. Where a connector cannot capture a relationship, OpenMetadata supports adding table- or column-level lineage explicitly.

Day-to-day, this gives a finance analyst, an operations manager, and a data engineer a common place to find the same product and verify its definition, owner, access route, and dependencies. The publishing partner remains responsible for the product’s quality, refresh cadence, and schema changes. Databricks runs the pipelines and enforces data access, while OpenMetadata gives consumers the context to use the product and gives stewards a place to manage its documentation, ownership, and lineage.

Databricks and OpenMetadata partner data fabric

Solid arrows show data flows and user access paths. Dotted arrows show metadata and lineage flowing into OpenMetadata.

Cross-organizational working group that coordinates how partners publish, document, share, and maintain their data products.

• Set publication standards. Agree on what every partner product should document, such as its purpose, accountable owner, business definitions, refresh cadence, quality expectations, and access route.

• Coordinate access and stewardship. Define who approves partner access, who implements it in Databricks, and who maintains the corresponding OpenMetadata entry. Each publishing partner can remain accountable for its own product.

• Resolve cross-partner issues. Assign owners and track fixes for missing lineage, conflicting definitions, failed refreshes, or schema changes that could disrupt another partner’s reports or pipelines.