Run pipelines on a schedule

Scheduled builds

Overview

Scheduled builds run at predefined intervals to materialize and test the assets in your data pipelines. You can construct a DAG using the y42 build command and set its execution frequency.

Schedule a build

Add a scheduled build

YAML configuration files

If you create a new scheduled build in Data Catalog mode, Y42 automatically generates a YAML file with the necessary configurations. In Code Editor mode, you have the option to modify existing scheduled builds or define new ones. Any changes you make will be reflected in both development modes.

Name of the scheduled build

Turn the scheduled build on/off

Build command to execute

Schedule in cron expression, which is automatically generated if you create a scheduled build in Data Catalog mode.

my_orchestration.yml

_13
version: 2
_13
_13
orchestrations:
_13
- name: marketing_orchestration
_13
config:
_13
enabled: true
_13
command: y42 build -s +mrt_sales
_13
cron: 0/30 * 1/1 * *
_13
- name: hubspot_orchestration
_13
config:
_13
enabled: false
_13
command: y42 build -s stg_ads+
_13
cron: 0/30 * 1/1 * *