BigQuery
Overview
This page contains the setup guide and reference information for the BigQuery source connector.
Output schema
The BigQuery source maintains the existing schema of your source database without introducing any changes. Depending on the destination configured in Y42, the destination schema can be altered. See the destination's documentation for more details.
Features
| Feature | Supported | Notes |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental Sync | No | |
| Change Data Capture | No | |
| SSL Support | No |
Data type mapping
| BigQuery Type | Resulting Type | Notes |
|---|---|---|
BOOL | Boolean | |
INT64 | Float | |
FLOAT64 | Float | |
NUMERIC | Float | |
BIGNUMERIC | Float | |
STRING | String | |
BYTES | String | |
DATE | String | In ISO8601 format |
DATETIME | String | In ISO8601 format |
TIMESTAMP | String | In ISO8601 format |
TIME | String | |
ARRAY | Array | |
STRUCT | Object | |
GEOGRAPHY | String |
Getting started
Requirements
For utilizing the BigQuery source, ensure you have the following:
- A Google Cloud Project with BigQuery enabled
- A Google Cloud Service Account with the "BigQuery User" and "BigQuery Data Editor" roles within your GCP project.
- A Service Account Key for authentication into your Service Account
See the setup guide for more information about how to create the required resources.
Service account configuration
Data syncing from BigQuery requires a service account (opens in a new tab) with granted permissions for BigQuery User and BigQuery Data Editor roles. These roles authorize you to execute BigQuery jobs, write to BigQuery Datasets, and access table metadata. For easier permission management and auditing, we suggest creating a dedicated Service Account. However, if an existing Service Account already has these permissions, you may use it instead.
To create a Service Account, follow the steps provided in GCP's Creating a Service Account (opens in a new tab). After creating the Service Account, remember to note down its ID for future role assignments. The Service Account IDs typically look like <account-name>@<project-name>.iam.gserviceaccount.com.
Next, add your newly created Service Account to your Google Cloud Project as a member with the BigQuery User role. Follow Granting Access (opens in a new tab) instructions provided by Google for this task. The Service Account ID you just created is the member's email address.
At this stage, you should have a service account granted with the BigQuery User project-level permission.
Service account key creation
Service Account Keys serve for authentication as Google Service Accounts. To utilize the permissions you accorded to the Service Account in the previous step, you'll need to generate its Service Account Keys. For more details about Keys, see Google documentation (opens in a new tab).
Follow the guide on Creating and Managing Service Account Keys (opens in a new tab) for key creation. Currently, we only support JSON Keys, so ensure your key is in this format. After creating the key, download it immediately as Google only permits viewing its content at this point. Upon successful configuration of BigQuery as a source, remove this key from your local machine.
Setting up the BigQuery source
With the requirements fulfilled, you can now proceed to configure BigQuery as a source in the UI. You'll require the following information for this setup:
- Project ID
- Default Dataset ID [Optional]: This is the schema name if only one schema is required. It significantly accelerates source discovery operations.
- Credentials JSON: This is the content of your Service Account Key JSON file
Once BigQuery is configured as a source, make sure to delete the Service Account Key from your computer.