Setting Advanced Configurations

Setting Source-Level Configurations

Following are the steps to set advanced configuration for a source:

1. Click the Data Catalog menu and click Ingest for the required source.

NOTE For an already ingested table, click the View Source button.

2. Click the Source Setup tab and scroll down to the Advanced Configuration section.
3. Click Add Configuration.
4. Enter key, value, and description. You can also select the configuration from the list that appears.

5. Click Save.

LIMITATION In Databricks persistent clusters, to set spark configurations on cluster level, the user will need to set the spark configurations on cluster and then restart the cluster with these configurations set. This is limitation from Databricks side.

Setting Table-Level Configurations

Following are the steps to set advanced configuration for a table:

1. Click the Data Catalog menu and click Ingest for the required source.

NOTE For an already ingested table, click View Source, click the Tables tab, click Configure for the required table and click the Advanced Configuration tab.

2. Click the Configure Tables tab, click the Advanced Configuration tab and click Add Configuration.
3. Enter key, value, and description. You can also select the configuration from the list that appears.

LIMITATION In Databricks persistent clusters, to set spark configurations on cluster level, the user will need to set the spark configurations on cluster and then restart the cluster with these configurations set. This is limitation from Databricks side. Setting spark configurations as advanced config will not work in case the compute is databricks-persistent cluster.