Configuring a Table

General Configuration

To configure a table, perform the following steps:

1. Click the Configure Tables tab.
2. Enter the ingestion configuration details.
3. Click Save.

Ingestion Configurations

SQL Queries

Select Query: The additional inputs required with a select clause, like optimization hints or lock changes. The default value is SELECT ${columnList} FROM <schemaName>.<tableName>, where, ${columnList} is the placeholder for all the columns to be queried and <schemaName> and <tableName> will be pre-filled with the appropriate values. This option is available only for Generic JDBC sources.

NOTE Ensure to use the correct SQL Query syntax as per the DB used.

Target Configuration

Optimization Configuration

Table Schema

You can view the following pages under table schema:

• Edit Schema
• Sample Data
• Onboarding Pipeline
• Lineage
• Audits

Editing Table Schema

You can edit the target datatype of columns using the following steps:

1. Click the Data Catalog menu and click View Source for the required source.
2. Click the Configure button for the required table.
3. Click the Table Schema tab and edit the target data type for required columns. The Databricks metastore table will be created with the specified datatypes.
4. The Options field allows you to edit the precision and scale of the specific column.
5. You can select a relevant column and click Manage Column to Exclude, Include or Bulk Edit.

• Select Include if you want the column to be a part of the target.
• Select Exclude if you want the column to be excluded from the target.
• Select Bulk Edit to edit the precision and scale of all the columns.

NOTE The Include and Exclude options are available only for Snowflake and BigQuery targets.

6. Click Save Schema.

Following are the datatype conversions supported:

Viewing Sample Data

The Sample Data page includes the sample data of the source table.

Onboarding Pipeline

The Data page includes a visual representation of all the Databricks metastore tables.

Viewing Data Lineage

The Lineage page displays all the instances where the data of the table is being used.

Viewing Source Audits

To view Source Audits, perform the following steps:

1. Click the Data Catalog menu and click View Source for the required source.
2. Click the Source Setup tab and scroll down to the Audits section. The audit logs for the source is displayed.

Alternatively, click the Tables tab > Configuration > Table Schema > Audits to view the audits section.

NOTE This screen displays only the modified properties. If a property was left unchanged during the iteration, it will not be displayed as an update, but the property will still be stored.

Audit Logs

In source audit logs, following are some operators:

• $push - the operator displayed when an entity is added.
• $pull - the operator displayed when an entity is deleted.

In table audit logs, when updating column details, following logs are displayed:

• columns.<columnIndexInTable>.<key>:<UpdatedValue> - indicates single column update.
• columns:[<list of all the columns with details>]- indicates bulk column update.

Advanced Configurations

Following are the steps to set advanced configuration for a table:

1. Click the Data Catalog menu and click Ingest for the required source.

NOTE For an already ingested table, click View Source, click the Tables tab, click Configure for the required table, and click the Advanced Configurations tab.

2. Click the Configure Tables tab, click the Advanced Configurations tab, and click Add Configuration.
3. Enter key, value, and description. You can also select the configuration from the list displayed.

Partitioning of Temporary Storage

This feature is not being used in 5.2 pilot release.

Subscribers

You can add subscribers to be notified for the ingestion at the table level.

Click Add Subscriber, and enter the following fields:

1. Emails: The email ID of the subscriber.
2. Notify Via: The means of notification. Available options are Email and Slack.
3. Notify For: The jobs for which subscribers must be notified.

Configuring a Table with CDW Onboarding

With the source metadata in the catalog, you can now configure the table for CDC and incremental synchronization. There are two types of Cloud Data Warehouse: Snowflake and BigQuery. Click the Configuration link for the desired table.

General Configurations (Snowflake)

In the General Configurations tab, provide the following Ingestion Configuration details:

General Configurations (BigQuery)

NOTE

• If the combination of schema name, database name and table name are same across different tables in the same environment, an error appears showing the combination of the three is conflicting with another table.
  • For Snowflake, we check the combination of table name, schema name and database name
  • For BigQuery, we check the combination of dataset name and table name.
  • For all other environments, we check combination of schema name and table name.