Publish assets

Publish assets

Overview

With Virtual Data Builds, you can manage your data environment by turning branches into virtual environments. This functionality offers flexibility and control over which branches to publish, and it allows for seamless integration with datasets and schemas in Bigquery and Snowflake.

Turning branches into data warehouse environments

Selecting branches to publish

By default, the main branch is automatically published. To publish additional branches, follow these steps:

  • Open Space Settings.
  • Navigate to Space settings > Branch environments > Choose a branch > More (...) > Edit > Publish dataset / schema.
  • Enable the publish option.
  • Specify the dataset or schema where you'd like to publish the data.

Input required for publishing to a dataset or schema.

Input required for publishing to a dataset or schema.

Once a branch has been added to Published Views, it will take approximately 1 minute to propagate.

Naming Convention for branches and assets

A new dataset (for Bigquery) or schema (for Snowflake) will be created using the following naming convention: y42__{{space_slug}}__{{branch_name}}.

Branch naming convention

Branch naming convention

The naming convention for the two types of assets materialized in the dataset/schema is as follows:

  • Model: {{model_name}}
  • Source: source__{{source_name}}__{{table_name}}

Asset naming convention

Asset naming convention

Creating assets based on the published branch

The assets will be created based on the state of the branch that you publish. The assets target the correct physical UUID tables, leveraging the internals of Virtual Data Builds.

Comparison between Y42 and BigQuery, illustrating the link between the asset in Y42, and the corresponding UUID in BigQuery.
BigqueryY42 application

Referencing assets

Y42 creates a view for every asset that has been built successfully, referencing the correct underlying UUID table of the DWH within the dataset/schema of the respective published branch.

If the asset is materialized as a table, the view will be replaced by a zero-copy clone (in Snowflake) or a table clone (in BigQuery). If the asset is materialized as a view, the published view will remain as a view.

Managing and controlling published assets

View Reference to the Latest Successful Build

  • The assets always reference the latest successful build of the respective branch (excluding builds that have failed tests).
  • Within the Y42 UI, you can find a redirection link straight to the view as part of the last successful build. This functionality only works if the branch is published as a virtual environment.
Last successful run of an asset

Last successful run of an asset

Unpublishing Assets

  • If certain assets shouldn't be published to the virtual branch environment of the respective branch, you can simply do so in the UI by hovering over the link and clicking on “Unpublish.”
Unpublishing an asset

Unpublishing an asset

  • Under the hood, this modifies a property called publish_view: boolean, which can also be edited directly in code mode.
models:
- name: my_first_dbt_model
description: "A starter dbt model"
config:
y42_table:
publish_view: false
columns:
- name: id
description: "The primary key for this table"
tests:
- unique
- not_null
If the property publish_view is not available, it means that the asset is published by default.

This feature is particularly useful if certain sources or models, e.g., staging models or raw sources, shouldn't be published for downstream consumption.