Onboarding Data from ORC

The Infoworks ORC Ingestion feature allows users to migrate data from Hadoop/Hive to Databricks.

Infoworks supports ORC files that have been ingested by Infoworks or created by Infoworks transformations in the source data lake.

Setting a ORC Source

For setting a ORC source, see Creating a Source. Ensure that the Source Type selected is ORC.

Configuring a ORC Source

For configuring a ORC source, see Setting Up a Source.

ORC Configurations

• Select either of the following storage locations depending on where the files are stored: Databricks File System (DBFS), Cloud Storage.

Databricks File System (DBFS) is a distributed file system mounted into a Databricks workspace and available on Databricks clusters. DBFS is an abstraction on top of scalable cloud object storage. For more details, see Databricks Documentation.

Infoworks allows you to access files from cloud storage, ingest the data, and perform analytics on them. Cloud storage refers to data stored on remote servers accessed from the internet, or cloud.

For preparing data for ingestion from Databricks File System (DBFS), enter the following:

• Source Base Path: The path to the base directory in DBFS where all the ORC files to be accessed are stored. The other files are relative from this path. For example, if the file is stored in iw/filestorage/ingestion/sample in DBFS, the base path refers to iw/filestorage/ingestion.

Infoworks allows you to access files from cloud storage, ingest the data, and perform analytics on them. Cloud storage refers to data stored on remote servers accessed from the internet, or cloud.

For preparing data for ingestion Cloud Storage, enter the following:

• Cloud Type: The options include Azure Blob Storage (WASB) or Amazon S3.

Windows Azure Storage Blob (WASB), also known as Blob Storage, is a file system for storing large amounts of unstructured object data, such as text or binary data. For more details, see Azure Blob Storage Documentation.

For Azure Blob Storage (WASB), enter the following:

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services that provides object storage through a web service interface. For more details, see Amazon S3 Documentation.

For Amazon S3, enter the following:

Google Cloud Storage (GCS) is an online file storage web service for storing and accessing data on Google Cloud Platform infrastructure. In GCS, buckets are the basic containers that hold the data. For more details, see GCS documentation.

For GCS, enter the following:

For Azure DataLake Storage (ADLS) Gen2, enter the following:

File Mapping

This page allows you to map individual files stored in the file system. You can add tables to represent each file, crawl them, and map the file details based on the preview.

Step 1: Click the File Mappings tab and click Add Table.

Step 2: Enter the following file location details:

Step 3: Click Save.

Step 4: Select the table and click Crawl Schema.

The metadata will be crawled and the job status will be displayed.

Ingesting ORC Data

For ingesting a ORC source, see Onboarding Data.

$inline[badge,LIMITATIONS,warning]

• Infoworks does not support ORC file ingestion when column name has space characters.
• Infoworks does not support handling of non-standard Spark partition data, unless the ORC files are created by Infoworks.
• Infoworks does not support reading data, for which the folder name in the partitioned table data is not of the part_col=val format.

Editing Schema

To edit the schema:

Step 1: Click the Details button for the required table.

Step 2: In the Edit Schema section, click the required column and perform the required changes. You can also add columns, upload schema, and perform bulk edit.

Additional Information

• If records are not adhering to schema and there is a datatype mismatch, the job will fail.