BigQuery

Overview

You can create a Google BigQuery Destination Connector to write to a Google BigQuery table.

To set up this Connector using a GCP Service Account Key, you will need a GCP Service Account that has access to the project(s) where the resources reside. To learn more about creating and managing service accounts within GCP, visit: https://cloud.google.com/iam/docs/creating-managing-service-accounts.

To set up this Destination Connector using Single Sign-On, your Google account will need access to the BigQuery table(s) you are connecting to.

The schema for this Destination Connector is defined by the BigQuery table schema.

Prerequisites

Required information:

• Single Sign-On credentials OR Service Account Key with the proper privileges

• Project ID

• Dataset

• Table

Creating an BigQuery Destination Connector

Step 1: After selecting + New Connector, under the system prompt, click BigQuery.

Step 2: Provide a Connector Name.

Step 3: Select Destination Connector.

Authentication

There are two options for authentication: Single Sign-On and Service Account. You can choose to authenticate through your Google account using single sign-on, or by providing a Google Cloud Platform service account JSON key.

• Step 4a: To authenticate using Single Sign-On, you will be prompted to sign into your Google Account after entering the required information and clicking Grant Access

• Step 4b: To authenticate using Service Account, you will be prompted to enter the Service Account Key. Provide the Service Account JSON key for the account you wish to connect to.

  • Service accounts associated with a BigQuery Destination connector will need Data Editor (read/write) privileges in order to successfully establish a connection.

Creating a Service Account Key in the Google Cloud Console

1. To create a Service Account JSON key, first navigate to the Service Accounts page in the Google Cloud Console.

2. Click the project dropdown in the top navigation bar to view all of your projects, choose the project you want to create a service account key for, and then click Open.

3. Find the row of the service account that you want to create a key for. In that row, click the More button, and then click Create key.

4. Select the JSON Key type and click Create

Step 5: Project ID

1. To find the Project ID, first select the project dropdown in the top navigation bar to view all of your projects.

The project dropdown can be found in the the upper left of Google Big Query page

2. Find the corresponding ID for the project, copy the Project ID and paste it into the Project ID field.

The Project ID in this example is example-project-365122

Dataset

Provide the name of the dataset that contains the table you want to connect to.

1. To find the dataset information, first Navigate to the BigQuery page in the Cloud Console.

2. In the navigation menu, click SQL Workspace

1. In the Explorer panel, expand the project name to see the datasets in that project. Copy the dataset name and paste it into the Dataset field.

For more information about creating a new Dataset, visit: https://cloud.google.com/bigquery/docs/datasets

Table

Step 1: Provide the name of the table you want to connect to. To find the table name, expand the dataset from the step above to see all of the associated tables, and copy the table name and paste it into the Table field.

Advanced Options

Validation Webhook

We support the use of Validation Webhooks to prevent bad data from being written to your systems, adding another layer of protection to the built-in validations that Osmos provides. The Webhook URL can be posted here.

For more information on Validation Webhook configuration, see Server Side Validation Webhooks

Connector Options

The connector can be deleted, edited and duplicated.

Duplication

To save time, the connector can be duplicated. This new connector needs to be named and can be edited, as needed.