Fivetran

Trigger and manage Fivetran Connector syncs.

Trigger Fivetran syncs out of Y42.

Overview

Y42 allows you to trigger and manage your Fivetran Connectors. Fivetran Source serves as a bridge between your Fivetran Connector and the Fivetran-produced tables within your data warehouse.

Fivetran Source

When a build is triggered, Y42 initiates a Sync request to Fivetran API, then monitors Sync status updates. Upon a successful sync, Y42 creates views for each updated Fivetran table, and subsequently triggers builds for any downstream assets, including data tests and models.

Requirements and prerequisites

Before setting up a Fivetran Source, you have to setup a Fivetran Connector (opens in a new tab) and initiate the first Sync. Your Fivetran Connector needs to be run at least once before the Connector can be triggered by Y42.

You will need to provide the following details during the setup process:

Connector ID: A unique identifier generated by Fivetran for each Connector. This can be found under Connector > Setup.
API Key: Can be generated under User > API Key.
API Secret: Can be generated under User > API Key.
Project / Database: The project / database of the Fivetran generated dataset / schema.
Dataset / Schema The dataset / schema of the Fivetran generated tables.

Set up guide

Create a Fivetran Source.

You can choose to use dbt packages for Fivetran or set up your source without them.

A. Create a Fivetran Source using dbt packages for Fivetran.

Y42 facilitates the use of community-maintained dbt packages for Fivetran (opens in a new tab). To install these packages, open the packages.yml file located in the root directory in code mode. Then, add the following entry to the yml file:


_10  - package: fivetran/package_name
_10    version: x.xx.x

Copy the name and version from the latest package description page, like the Google Ads package (opens in a new tab). For instance, add the following to the yml file:


_10  - package: fivetran/google_ads
_10    version: 0.10.1

Next, install the packages by pressing CMD / CTRL + K and selecting Install package dependencies. This will create a new "dbt_packages" folder in the catalogue, containing a folder for each installed package, along with all its dependencies.
Locate the source folder within a package folder under dbt_package folder in the Code view. The community maintaining Fivetran dbt packages typically names source folders with a "source" suffix (e.g., google_ads_source or hubspot_source).

Finally, click on the desired source and select "External Fivetran." This links the source in the package with your Fivetran credentials, enabling API calls to the Fivetran API for initiating data syncs.

B. Create a Fivetran Source without importing packages.

In the asset creation pop up, select "External Fivetran" and enter a unique name for the Fivetran Source.

Add a new Fivetran source.

Create a new connection or use an existing connection.

On the source setup page, on the Authorization panel, select from existing secrets or create a new one.

Create new Fivetran connection.

To create a new secret, enter a unique name and provide Fivetran API key and secret (opens in a new tab).

Generate a new Fivetran API key.

Declare the Fivetran Connector ID, database and schema of your Fivetran tables.

Open the settings tab and enter:

Fivetran Connector ID. Can be found under Connector > Setup section.
Project / Database and Dataset / Schema of your Fivetran tables. Those are the properties of your DWH where Fivetran created tables after the initial sync.

Verify selected tables and columns.

Open the schema tab and verify that you are referencing the correct tables.

Commit your changes.

Considerations

An individual Fivetran table build will automatically trigger builds for all tables of the Fivetran Connector. Fivetran does not allow individual table syncs.
Failed data tests on Fivetran tables will only stop the DAG and not prevent any updates on the tables themselves.

Stripe Python sources