v6.2.0 - Draft

Date of Release: August 2025

• Workflow Versioning Introduced: Manage and track changes to workflow configurations over time, enabling better collaboration, stability, and traceability. Refer to auto$ for details.
• Expanded Audit Coverage: Auditing is now available for topic mapping in Kafka ingestion and Confluent Cloud, file mapping in JSON ingestion, and table group configurations, enabling improved traceability across all onboarding workflows.
• Pipeline & Pipeline Group Audits: Audit logging is now available for both individual pipelines and pipeline groups, enabling visibility into user actions and configuration changes.
• Visual Job Progress for Pipeline Groups Introduced: The new Job Progress tab offers an intuitive, node‑based graphical layout that displays pipeline execution flow, status, version, and dependencies.
• Enhanced Parameterization Support: You can now use workflow version parameters in the Recipient Email Addresses, Subject, and Message fields.
• Source Catalog Filtering: Introduced a source catalog field to limit assets in Hive and Delta Metasync sources within Unity environments.
• Refresh Token Expiry Enforcement: To enhance security and manageability, the system now enforces refresh token expiry with options for administrative configuration and automated notifications.
• Email Summary Notifications: Introduced email summary notifications for workflow runs, with options to attach failed task logs and a summary. These can be enabled via checkboxes in the 'Send Notification' task properties.
• Cache Manager: This new service improves the performance and scalability of data preview operations by separating cache management from core ingestion and transformation workflows. It stores encrypted preview data with a one-hour TTL and uses Hazelcast as the default caching solution. The service also supports configuration to use a Redis cache endpoint as an alternative.
• Spark 2 Deprecated: Starting from version 6.2.0, support for Spark 2 has been deprecated.
• Spark on Kubernetes Support (Beta Feature): Infoworks now supports running Spark workloads natively on Kubernetes-managed clusters. Find detailed information in auto$.
• LDAP Test Connection Security Enhancement (Beta Feature): The approach to TLS certificate handling for LDAP test connections has been updated.

Enhancements

This section consists of the improvements in this release:

Resolved Issues

This section consists of the resolved issues in this release:

Known Issues

• Audit updates for sensitive entities such as S3 access/secret keys and WASB account keys are not captured.
• Incorrect preview data is observed when the filter condition is modified via the pipeline version parameter.
• Sync to Target and pipeline jobs may fail with SQL Server targets when the table name contains a reserved keyword.
• Sync to Target and pipeline build jobs with Snowflake targets may fail when Enable Schema Sync is enabled if columns are added or deleted after the first job run.
• In the onboarding flow, the target table name accepts duplicates when the case does not match.
• Ingestion fails in the Unity Catalog environment when the target catalog name contains a '-' character.
• Able to view information_schema and its tables for a metadata sync source in a Unity Catalog environment.
• Creation of a new generic JDBC extension fails when specifying the folder location.
• In preview data, SQL queries are ran by the previously selected profile.
• The target table in a SQL pipeline created through SQL import is unavailable as a reference table.
• Insert Overwrite mode is not supported for spark native targets in transformation pipelines.
• The SCD2 merge audit columns are not being updated correctly for referenced tables in Snowflake and Datalake environments.
• Segment load jobs are reporting incorrect row counts during ingestion in the Snowflake environment.
• Workflow parameters are incorrectly appearing as the Pipeline Parameters in the Pipeline Node, despite no parameters being defined.
• Preview Data in custom target node in a pipeline is not supported.
• Streaming jobs that have been stopped may show running state for the cluster job, users can verify that the job is actually stopped by observing that the number of batches run for that job does not increase after stopping it; more details here.
• Micro batches processing stop for streaming ingestion if the source stops streaming the data for Databricks runtime version 14.3.
• Pipeline build failing when read from merged/deduplicated table is selected.

Limitations

• After upgrading to version 6.2.0, failed workflow runs from earlier releases cannot be restarted due to the introduction of versioning support in workflows.
• Writing to the Iceberg uniform format is not supported when the table is partitioned by a TIMESTAMP column, as Iceberg does not support partitioning on raw timestamp fields during Delta-to-Iceberg conversion. Since the write flow involves writing to Delta first and then performing a reorg to Iceberg, any TIMESTAMP-based partitioning will cause the conversion to fail.
• Sync to target as Azure Cosmos DB or setting Azure Cosmos DB as a pipeline target is not supported in Azure Databricks environments with Unity Catalog enabled when using a shared mode cluster. This is a limitation of the Azure Cosmos DB connector.
• Pipeline node preview data is not supported when using clusters in Shared mode with Unity Catalog.
• Discrepancy while performing 'EDIT' configuration for 'Authentication Type' for a persistent cluster : The user is unable to change the authentication type for an existing cluster. Note: The cluster creator cannot be changed once created. Updating authentication details with different user credentials will not affect the creator. To change the creator, the cluster must be deleted from the Databricks console and recreated from Infoworks.
• When the table is ingested using Databricks 14.3 and in the pipeline when we check for preview data either with 11.3 or 14.3, API gets timed out.
• Streaming is not supported on a shared cluster.
• If the target node properties (e.g., target table name or target table base path) are changed after a successful pipeline build, DT will not treat the modified table as a new table. Note: If a user needs to update the target node properties, they must delete the existing target node and configure a new one in the pipeline editor.
• For TPT jobs running on shared cluster, it is user's responsibility to install TPT otherwise job will not work due to limitation from databricks.
• In a non-Unity Catalog environment, execution type Databricks SQL is only supported in DBFS storage.
• Target tables used in the SQL pipelines without a create table query, will not be available in the data models for use.
• Spark execution type does not support SQL pipelines.
• Jobs on the Databricks Unity environment fail with "Error code: FILE_NOT_FOUND_FAILURE." Refer here.

For Kubernetes-based installation, refer to Infoworks Installation on Azure Kubernetes Service (AKS).

For more information, contact support@infoworks.io.

Upgrade

For upgrading Azure Kubernetes, refer to Upgrading Infoworks from 6.1.3.x to 6.2.0 for Azure Kubernetes.

PAM

Please refer to Product Availability Matrix (PAM).