Managing Secrets

Overview

Service Authorization

To avoid password-based or user-based authentication, service authentication can be used. These are identities used by applications and services to access specific resources, such as performing Databricks operations or accessing ADLS storage. These authentications can have a specific role, and tightly controlled permissions to access only required resources.

NOTE Infoworks currently supports Azure AD service principal and managed identity

External Keyvault

To avoid storing sensitive information in a metadb, such as password of sources or access token of Databricks workspace, you can use External Keyvault to store this sensitive information and Infoworks will retrieve this information when required.

NOTE Infoworks currently supports Azure Keyvault.

Architecture

Service Authorization

The below-given image illustrates the architecture of Service Authentication in Infoworks.

IWX Edge Node: This is a VM where Infoworks services are deployed. The Config Service on Edge Node is the only client for the service authentication. All other services will connect to the config service when any access token is required.

Config Service will maintain a cache of Service Authentication Access Tokens in memory to avoid multiple requests to the service in a short timeframe.

External Keyvault

The below-given image illustrates the architecture of External Keyvault Usage in Infoworks.

IWX Edge Node: This is a VM where Infoworks services are deployed. The Config Service on Edge Node is the only client for the external keyvault. All other services will connect to the config service when any secret value is required.

Config Service will maintain a cache of Secret Value in memory to avoid multiple requests to the external keyvault in a short timeframe.

keyvault_secret_cache_interval is the config to control cache expiry duration in the config service. The default expiry duration is 10 minutes (value should be provided in milliseconds).

Prerequisites

Creating Service Authentication in Azure Portal

An Azure service principal is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level.

To create a Service Principal in the Azure portal, refer to Creating a Service Principal.

NOTE Infoworks currently supports Service Principal authentication through Application Secrets. For more information, refer to Creating a New Application Secret.

Managed Identities provide an automatically managed identity in Azure Active Directory (Azure AD) for applications to use when connecting to resources that support Azure AD authentication. Applications can use managed identities to obtain Azure AD tokens without having to manage any credentials.

To create a Managed Identity in the Azure portal, refer to Creating a Managed Identity.

Assigning Authentication in Azure Portal

Assigning Authentication for Azure Databricks

For Azure Databricks, assigning authentication is a two-step process:

Step 1: To assign Azure role (such as Contributor) for the Service Principal on the Databricks resource refer to Managing Roles.

Step 2: To assign authentication for Databricks, refer to refer to Assigning Authentication in Azure Databricks.

To assign authentication for Storage, refer to Assigning Storage Access.

Assigning Authentication for Azure ADLS

To assign authentication for Storage, refer to Assigning Storage Access.

Creating Keyvault in Azure Portal

To create a Keyvault in the Azure portal, refer to Creating Azure Keyvault.

NOTE Infoworks only supports Vault Access Policy as the access configuration for Azure Keyvault.

Assigning Keyvault Access in Azure Portal

To assign a Keyvault access policy, refer to Assigning Keyvault Access.

NOTE Infoworks requires the GET and LIST permissions/policies on Secrets in Keyvault.

Copying Keyvault URL from Azure Portal

To copy the Keyvault URL, which is generated after the keyvault is created.

Step 1: Log in to Azure portal and navigate to Key Vaults section.

Step 2: Click the respective Keyvault, and copy Vault URI.

Adding Service Authentication

Adding Service Principal/Managed Identity

To add Service Principal/Managed Identity:

Step 1: Go to Admin > Manage Secrets. The Secrets Store Authentication tab appears.

Step 2: Click Add Secrets Store Authentication.

Step 3: Provide the following details.

Step 4: Click Save.

Adding Secret Store/Keyvault

To add Secret Store/Keyvault:

Step 1: Go to Admin > Manage Secrets.

Step 2: Click Secret Stores tab and click Add Secrets Store.

Step 3: Provide the following details.

Step 4: Click Save.

Creating Secrets

A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, or cryptographic keys.

Once the Secrets Store has been set up, you can create Secrets retrieve sensitive information as per the requirements.

Prerequisites

Creating Secrets in Azure Key Vault

To create a secret in Azure Key Vault, refer to the Adding a Secret in Key Vault.

Creating Secrets in Infoworks

To create secrets:

Step 1: Go to Admin > Manage Secrets.

Step 2: Click Secrets tab and click Add Secret.

Step 3: Provide the following details.

Step 4: Click Save. The Secret can now be used to retrieve passwords and other information.

Creating Authentication Services in Infoworks

Adding Service Principal/Managed Identity for Service

To add Service Principal/Managed Identity for Services:

Step 1: Go to Admin > Manage Secrets. The Authentication Service tab appears.

Step 2: Click Add Authentication Service.

Step 3: Provide the following details

Step 4: Click Save.