v5.2.0

Date of Release: May 16, 2022

New Features and Enhancements

Infoworks now automates onboarding of data directly to Snowflake and supports data transformation in Snowflake (SQL pushdown). Infoworks can orchestrate data management across hybrid environments including Snowflake and data lakes.

Component: Platform

• Databricks Support on GCP: Infoworks now supports Databricks on GCP. The Databricks version supported is 9.1 LTS across all environments - Azure, AWS and GCP. For more details, see Configuring Infoworks with Databricks on GCP.
• Configuring a Snowflake Data Environment: Infoworks enables you to easily configure and manage the Snowflake connection, data warehouses and related compute and storage resources needed to run data onboarding and preparation jobs. The associated Spark compute may be one of Azure Databricks, AWS Databricks or Databricks on GCP. For more details, see Configuring Infoworks with Snowflake.
• OAuth Support for Snowflake Environments: This feature allows you to configure OAuth as an authentication mechanism for Snowflake connectivity. Infoworks supports the OAuth service provided by Snowflake and Azure AD as an external authorization provider. For more details, seeConfiguring Infoworks with Snowflake.
• Support for Hybrid Deployments: Infoworks now supports the orchestration of data management tasks across data lake and data warehouse environments. An Infoworks domain may span multiple data environments. Infoworks also supports the migration of pipelines between environments, for example, from a data lake to a cloud data warehouse. For more details, see Managing Domains.

Component: Onboard Data

• Onboard Data directly to Snowflake: Infoworks now supports onboarding data directly to Snowflake from a variety of sources including RDBMS, files and SaaS systems. For more details, see Onboarding an RDBMS Source.
• Support for SCD on Snowflake Target in a Snowflake Environment: Snowflake onboarding now supports SCD options including SCD1 and SCD2. For more details, see Configuring a table.
• Support to Update Teradata JDBC driver from Outside Infoworks: Infoworks provides the support to update the latest Teradata JDBC driver file downloaded externally. For more details, see Upgrading to 5.2.0.
• Streaming Support in Snowflake Environment: Infoworks now supports onboarding data from streaming sources, such as, Kafka and Confluent to Snowflake. For more details, see Onboarding Data from Kafka and Onboarding Data from Confluent Cloud.
• Support for Adding Columns to CSV Table: Infoworks now supports adding columns to the csv table after schema is inferred. This feature allows you to define source columns or custom target columns which might be useful in the downstream of the data. For more details, see Onboarding Data from Structured Files.
• SSH Key based Authentication for Ingestion using SFTP: This feature supports authenticating CSV, JSON, and Mainframe data files sources using SSH keys when onboarding data from a remote server via SFTP. For more details, see the Onboarding Data page for JSON, Structured files, and Mainframe Data Files.
• Support for reading the CSV data from ADLS Gen 2: Infoworks now supports reading the CSV data from ADLS gen 2 in addition to Blob. For more details, see Onboarding Data from Structured Files.

Component: Prepare Data

• Snowflake Pushdown: Infoworks now supports automated transformation pipelines that runs in Snowflake warehouses. Infoworks automates the generation and execution of the required SQL code (SQL pushdown). Transformed data may be written to existing Snowflake tables in append, overwrite, or merge modes. For more details, see Snowflake Node Configurations.
• Preview SQL: This feature allows you to preview the SQL code to be executed in order to transform data in Snowflake. For more details, see Node Settings.
• SQL Import to Pipelines: Infoworks supports SQL import into pipelines that run in a Snowflake environment.

Known Issues

Resolved Issues

Installation

For the Installation procedure, see Infoworks Installation on Azure , Infoworks Installation on AWS, and Infoworks Installation on GCP.

For more information, contact support@infoworks.io.

Upgrade

For upgrading from lower versions to 5.2.0, see Upgrading to 5.2.0.

PAM

For the PAM, see Product Availability Matrix.