BigQuery
Chord uses a service account in your Google Cloud project to copy the BigQuery tables you choose into a Cloud Storage bucket you own, as Parquet files. Chord then reads the files from that bucket.
You create the account, grant it four roles, and send Chord the account details, a JSON key and a list of what to share.
Before you start, you need a Cloud Storage bucket that Chord can read. The bucket must be in the same location as the BigQuery datasets you share.
Sharing GA4 export data? Use the separate GA4 guide instead. It covers the analytics_<property_id> dataset.
Pick one way to set this up, not both.
Option A uses the Google Cloud console.
Option B runs the same steps with the gcloud CLI.
Both end in the same place. Then go to Choose what to share.
Option A: Google Cloud console
Create the service account
Use the project that holds the datasets you want to share.
- In the Google Cloud console, open the Create service account page and select that project.
- Enter a name, for example chord-bigquery. Google builds the account ID from it, and you cannot change the ID later.
- Click Create and continue, then Done. You will grant the roles in the next section, so you can skip the role step here.
You need the Create Service Accounts role (roles/iam.serviceAccountCreator) on the project. To grant roles you also need Project IAM Admin (roles/resourcemanager.projectIamAdmin). A new service account can take a minute or more before it is usable.
Grant the roles
Grant each role at the narrowest level Google allows. You will need the service account's email, for example [email protected].
On the project.
BigQuery Job User and BigQuery Read Session User can only be granted on a project, folder or organization, never on a single dataset.
- Open IAM & Admin → IAM and select your project.
- Click Grant access.
- Enter the service account email under New principals.
- Add the roles BigQuery Job User and BigQuery Read Session User, then click Save.
On each dataset.
BigQuery Data Viewer must be granted on every dataset you share. Repeat these steps for each one.
- In the Google Cloud console, open BigQuery.
- In Explorer, expand your project, click Datasets, and select the dataset.
- Click Sharing → Permissions, then Add principal.
- Enter the service account email.
- In Select a role, choose BigQuery Data Viewer.
- Click Save, then Close.
On the bucket.
Storage Object Admin only needs to cover your Cloud Storage bucket.
- Open Cloud Storage → Buckets and click the bucket name.
- Open the Permissions tab and click Grant access.
- Enter the service account email under New principals.
- In Select a role, choose Storage Object Admin, then click Save.
Create a JSON key
- Open the Service accounts page and select your project.
- Click the email address of the service account.
- Open the Keys tab.
- Click Add key, then Create new key.
- Select JSON as the key type and click Create.
The key file downloads once and Google cannot show it again.
The JSON key is a secret. Send it to Chord through a secure channel, never by email or Slack. Delete your local copy once Chord confirms the connection works.
Option B: gcloud CLI
Skip this section if you used Option A. It does the same thing from the command line.
Set these values first.
PROJECT_ID=your-project-id
BUCKET=your-bucket-name
SA=chord-bigquery
SA_EMAIL=$SA@$PROJECT_ID.iam.gserviceaccount.comCreate the account.
gcloud iam service-accounts create $SA \
--project=$PROJECT_ID \
--display-name="Chord BigQuery replication"Grant the two project-level roles.
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$SA_EMAIL" --role="roles/bigquery.jobUser"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$SA_EMAIL" --role="roles/bigquery.readSessionUser"Grant Data Viewer on each dataset you share. Run this in the BigQuery query editor, once per dataset.
GRANT `roles/bigquery.dataViewer`
ON SCHEMA `your-project-id`.your_dataset
TO "serviceAccount:[email protected]"Grant Storage Object Admin on the bucket.
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
--member="serviceAccount:$SA_EMAIL" --role="roles/storage.objectAdmin"Create the key.
gcloud iam service-accounts keys create chord-bigquery-key.json \
--iam-account=$SA_EMAILThe key file chord-bigquery-key.json is saved in your current folder. Treat it as a secret, as in Option A.
Choose what to share
List every table or view you want Chord to copy. Use one dataset.table per line and say whether each one is a table or a view. For example:
sales.orders (table)
sales.customers (table)
marketing.campaign_summary (view)Share base tables wherever you can. The notes below explain why.
Tables and views.
Chord copies a table with a BigQuery extract job. Google does not charge a processing fee for extract jobs.
An extract job cannot export a view. To copy a view, Chord has to run a query, and BigQuery bills that query to your project.
Views that read other datasets.
Querying a view requires access to the data it reads. If a view reads tables in other datasets, grant the service account BigQuery Data Viewer on those datasets too. The exception is an authorized view, which has its own read access to the source dataset.
Views that read a wildcard table.
If a view reads a wildcard table such as events_*, share the underlying daily tables instead. Exports of these views have failed in practice.
Large views.
A view that sorts tens of millions of rows with ORDER BY can run out of query memory and fail. Share the base table instead, or remove the ORDER BY from the view.
Tables backed by Google Sheets.
If a table or view reads from a Google Sheet, share the Sheet itself with the service account email. Without that, the export fails.
Data that expires.
A table with an expiration date is deleted when it expires, along with its data. This is common in the BigQuery sandbox, where tables expire after 60 days by default. Check the expiration on anything you share. See Google's guide to table expiration.
Location.
Keep the bucket in the same location as the datasets you share. Google requires this for exports to Cloud Storage.
What to send Chord
- Service account email. For example [email protected].
- JSON key. The key file, through a secure channel.
- GCP project ID. The project that holds the datasets.
- Bucket name. The full path, starting with gs://.
- Datasets and tables. Your dataset.table list, with each one marked as a table or a view.
Chord configures the connection from these values. Chord refreshes the data on a schedule you agree with your Chord representative.
Rotating and revoking access
Rotation. Google recommends rotating keys regularly. Create a new key, send it to Chord, wait for Chord to confirm it works, then delete the old key. Google advises disabling a key before deleting it, because a deleted key cannot be restored.
Revoking access. Deleting the key cuts off Chord's access, and replication stops until you send a new one. To delete a key, open the service account, open the Keys tab, and click Delete next to the key. Deleting a key does not cancel short-lived credentials already issued from it. To cut those off as well, disable or delete the service account.
Troubleshooting
Why can't I create the JSON key?
Your organization may enforce the iam.disableServiceAccountKeyCreation policy. Google turns it on by default for organizations created on or after May 3, 2024. An organization policy administrator can exempt your project. They tag the project not_enforced and update the policy so it applies to everything except tagged resources. This needs the Organization Policy Administrator, Organization Viewer and Tag Administrator roles. See Google's guide.
Why does Chord report a permission error?
Check that all four roles are in place. The two BigQuery roles must be on the project, Data Viewer on every dataset you share (including datasets that your views read from), and Storage Object Admin on the bucket.
Why does one table or view fail while the others work?
Check the notes in Choose what to share. The usual causes are a view that reads a wildcard table, a view or table backed by a Google Sheet that is not shared with the service account, or a view that runs out of query memory.
Need help? Contact your Chord representative.