Working with assets
Assets are the buildings blocks of data pipelines. By adhering to a stateful and declarative asset-based paradigm, Y42 ensures that your data pipelines are not only scalable but also easy to maintain, with built-in governance and observability guardrails.
Y42 assets
Types of assets
Y42 offers a versatile range of assets, each designed to fulfill specific roles within your data pipeline architecture:
| Asset type | Capability |
|---|---|
| Reference source | Read or reference existing tables from your data warehouse. |
| CData connector | Ingest data from hundreds of pre-built connectors, powered by CData. |
| Airbyte connector | Similar to CData connector, powered by Airbyte (open source). |
| Python source | Ingest data using Python cloud functions. |
| Fivetran source | Trigger syncs in Fivetran and reference the generated tables. |
| CSV seed | Store CSV files and sync them with the data warehouse. |
| dbt model | Transform data with modular SQL models. |
| Python model (Closed beta) | Transform data and run machine learning workloads on data platforms using Python. |
| Snapshot | Capture snapshots of data for managing slowly changing dimensions (SCD Type 2). |
| Exposure | Document downstream data consumers or tools. |
| Python action | Sync data pipelines with external systems or workflows using Python cloud functions. |
Natively compatible assets
Despite their varied capabilities, all assets share a standard configuration schema, including metadata fields. This design principle guarantees natively compatible assets that can be easily managed and integrated. For instance, from dbt models, Airbyte connectors to Python cloud functions, all assets can be selected and executed using a common build command, streamlining the pipeline construction process.
Learn more about asset properties here.
Managing assets
Y42 lets you create, modify and delete assets through both a graphical interface and a code editor, powered by the open-source distribution of VS Code. At their core, assets are defined by SQL or YAML configuration files, which are stored and version-controlled within a Git repository. This dual-interface approach ensures that assets can be created and integrated into existing pipelines efficiently, catering to both GUI-oriented and code-savvy users.
Learn more about different development modes here.