Project configuration
Y42 project configuration.
Overview
A Y42 project is a collection of SQL and YAML code files. The SQL files define data transformations, while the YAML files specify configurations. These files can be further organized into subdirectories, according to your desired structure.
Project structure
When you create a space, Y42 initializes a Git repository with the default folder structure shown below. All top-level folders must be present in order for the project to be successfully compiled. However, you can customize the internal folder structure of each top-level folder to best suit your organizational needs.
Project config: Define your project's metadata and table settings.
Sources: Load and extract your raw data from external sources.
Models: Transform your data, so that it is ready for analysis.
Tests: Run tests to check whether your tables are correct.
Orchestrations: Build your data pipelines in an automatic workflow.
Alerts: Send notifications when tests fail during orchestrations.
Selectors: Specify builds of the data pipeline that are more advanced.
Snapshots: Capture states of data over time.
Analysis: Explore your data with ad-hoc transformations.
Macros: Write reusable code for your SQL statements.
Packages: Import prewritten code from others to your project.
Gitignore: Choose files or directories that shouldn't be tracked with git.
And there we have it, our data pipelines defined in a unified directory.
Project configuration file
The dbt_project.yml file serves as the project-level configuration file. It is the only file that needs to be defined, at minimum, and must be placed in the project's root directory. Note that you can access this file, along with all other .yml configuration files, exclusively in Code Editor mode.
Key components of the project configuration file:
- Project name
- Version
- Directory structure
- Asset configurations
FAQ
Can I import an existing dbt project into a Y42 space?
Y42 extends several of dbt's built-in functionalities. To leverage these enhancements, you can modify an existing dbt_project.yml file to follow the Y42 project configuration specification. Note that there are several resource properties that are not supported or are manifested in a different way.
Are there any differences between dbt yaml configurations and Y42 yaml configurations?
Y42 introduces additional attributes to the dbt YAML configuration specification. These extensions bring new functionalities such as ingestion, orchestration, alerts and publication.