v5.5.0

Date of Release: September 2023

• Infoworks now supports EMR version 6.6 in addition to 6.2. For more information, refer to the "Type and Run Time Version" field in the Compute section. Also, note that support for 5.32 has been deprecated.
• Infoworks will support creation of multiple profiles in single snowflake environment. Users using snowflake environment will now have a choice to select profile during source, pipeline and pipeline group creation. For more details, please refer to the Table Schema section here.
• For sources created on Snowflake environments, time data type has been added. Additionally there is a textbox which takes the precision for time data type columns. The value of this is prefilled during metacrawl but it can be edited. Refer to Configuring a Table.
• Infoworks now supports access control when using Databricks clusters and jobs. Infoworks will accept access control field as json string and use the field as is in Databricks API calls for cluster creation and job submission. Access control must be enabled and configured as per Databricks documentation before usage in infoworks. For detailed information, please refer to the Access Control List section here.
• Now the user will have the option to synchronize pipeline export table schema with source table schema for the targets: BiqQuery Target Table, Snowflake Target Table, Postgres Target Table, Teradata Target Table, Oracle Target Table, Redshift Target Table, SQL Server Target Table, Azure Synapse Target Table. To know more, refer to the sections listed under Pipeline Targets.
• Support for storing and accessing client secret of service principal from Azure Key Vault.Now we can have two sets of service principal one for Azure key vault authentication and other for authentication of Databricks api whose client secret stored in azure key vault. Refer to Managing Secrets.
• With SLA configurations, user can now configure expected run time of job (eg: hours and minutes) on entity level (table_groups, pipeline, pipeline_groups, workflow) and get notified if job run exceeds configured run time. For more details, view pages: SLA Configuration and Criticality and Using Operations Analyst Dashboard.
• Now the SQL Import feature supports the DELETE and UPDATE queries. Refer to Importing SQL.
• Onboarding flow for SFDC sources modified to enable the ability to only onboard the required tables to the source. For more information, view Onboarding Data from SalesForce page.
• New read only role added to provide view only access of infoworks. For more details, check out the Editing a User section here.
• Infoworks now supports native idle timeout option for Dataproc and EMR persistent clusters (previously only available for Databricks).
• There are a few changes in the transitive access control of domains through the “Accessible Domains” functionality. Please review here for specific changes.
• The Databricks runtime version for 9.1 has been deprecated. More information can be found under the Compute section.
• There are changes in SAML ACS URL format to work with V3 APIs. Please check here for information.

$inline[badge,Resolved Issues,success]

This section consists of the resolved issues in this release:

$inline[badge,Known Issues,error]

The following section contains known issues that Infoworks is aware of, and is working on a fix in an upcoming release:

$inline[badge,Limitations,warning]

• Persistent Cluster : Cluster restart does not work reliably with inactivity-timeout set less than 10 minutes.
• For SQL pipelines, use of snowflake variables as a table identifier / table name not supported.
• BigQuery as a sync to target is not supported on EMR 6.6.
• Deploy mode as a Cluster is not supported on EMR and Dataproc.

$inline[badge,Installation,primary]

For Kubernetes-based installation, refer to Infoworks Installation on Azure Kubernetes Service (AKS).

For VM-based installation, refer to VM-Based Deployment.

For more information, contact support@infoworks.io.

$inline[badge,Upgrade,primary]

For upgrading from 5.4.2.x to 5.5.0 for Azure Kubernetes, refer to Upgrading Infoworks from 5.4.2.x to 5.5.0 for Azure Kubernetes.

For upgrading from 5.0 to 5.5.0 for VM, refer to Upgrading Infoworks from 5.0 to 5.5.0 for VM.

$inline[badge,PAM,primary]

The Product Availability Matrix (PAM) is available here.