Publish assets

Overview

With Virtual Data Builds, you can manage your data environment by turning branches into virtual environments. This functionality offers flexibility and control over which branches to publish, and it allows for seamless integration with datasets and schemas in Bigquery and Snowflake.

Turning branches into data warehouse environments

Selecting branches to publish

By default, the main branch is automatically published. To publish additional branches, follow these steps:

• Open Space Settings.
• Navigate to Space settings > Branch environments > Choose a branch > More (...) > Edit > Publish dataset / schema.
• Enable the publish option.
• Specify the dataset or schema where you'd like to publish the data.

Input required for publishing to a dataset or schema.

Naming Convention for branches and assets

A new dataset (for Bigquery) or schema (for Snowflake) will be created using the following naming convention: y42__{{space_slug}}__{{branch_name}}.

Branch naming convention

The naming convention for the two types of assets materialized in the dataset/schema is as follows:

• Model: {{model_name}}
• Source: source__{{source_name}}__{{table_name}}

Asset naming convention

Creating assets based on the published branch

The assets will be created based on the state of the branch that you publish. The assets target the correct physical UUID tables, leveraging the internals of Virtual Data Builds.

Comparison between Y42 and BigQuery, illustrating the link between the asset in Y42, and the corresponding UUID in BigQuery.

Referencing assets

Y42 creates a view for every asset that has been built successfully, referencing the correct underlying UUID table of the DWH within the dataset/schema of the respective published branch.

Managing and controlling published assets

View Reference to the Latest Successful Build

• The assets always reference the latest successful build of the respective branch (excluding builds that have failed tests).
• Within the Y42 UI, you can find a redirection link straight to the view as part of the last successful build. This functionality only works if the branch is published as a virtual environment.

Last successful run of an asset

Unpublishing Assets

• If certain assets shouldn't be published to the virtual branch environment of the respective branch, you can simply do so in the UI by hovering over the link and clicking on “Unpublish.”

Unpublishing an asset

• Under the hood, this modifies a property called publish_view: boolean, which can also be edited directly in code mode.

This feature is particularly useful if certain sources or models, e.g., staging models or raw sources, shouldn't be published for downstream consumption.