Error in accessing ADLS using Databricks CLI from Edge Node

Error in accessing ADLS using Databricks CLI from Edge Node

Problem description:When ADLS mount point is accessed using DBFS CLI, one may encounter the following error

infoworks@ifwdfndryp01:~ dbfs ls dbfs:/mnt/ifwdatafoundry

Error: b'{"error_code":"IO_ERROR","message":"GET https://adlsstorage.dfs.core.windows.net/ifwdatafoundry?resource=filesystem&maxResults=5000&timeout=90&recursive=false\\nStatusCode=403\\nStatusDescription=This request is not authorized to perform this operation.\\nErrorCode=AuthorizationFailure\\nErrorMessage=This request is not authorized to perform this operation.\\nRequestId:4fa16227-v0123-007f-2424-2dbe46000000\\nTime:2020-05-18T15:23:27.4766665Z"}'

Root cause: ADLS storage is restricted to private/selected storage

Resolution: This issue can be at the subnet level or the private network. To identify use the traceroute for the ADLS storage from the VM.

Command : traceroute [storage name]

1. Once, you are ready with your Azure Databricks setup, you need to whitelist your virtual network and all its subnets in the Azure Data Lake Gen2. You can do this in the Firewalls and Virtual networks section from your management pane in Azure Portal.

Click on ‘Add existing virtual network’ and select your subscription and VNET in which your Azure Databricks is deployed. Make sure you include all the subnets.

2. The Azure VPN will have some restricted IP for azure data bricks to access the storage. Use the user-defined route settings for azure data bricks from the link below. Add the IP on Firewall and virtual networks for the ADLS storage under the address range.

Example:- For 'West US' add 40.118.174.12 and '40.83.178.242' in the address range

<https://docs.microsoft.com/en->us/azure/databricks/administration-guide/cloud-configurations/azure/udr#control-plane-nat-and-webapp-ip-addresses