v5.4.1
Date of Release: March 2023
This section consists of the new features and enhancements introduced in this release. |
Support for GKE: Infoworks’ control plane may now be installed and used on Google Kubernetes Engine (GKE). The deployment model enables failover of Infoworks pods, providing better availability and scalability when a large number of concurrent jobs and workflows are executed. For more details, refer to Installing Infoworks on Google Kubernetes Engine (GKE).
Autoscaling using Kubernetes: Infoworks now supports autoscaling when the control plane is deployed on Kubernetes, providing for significantly increased scalability for enterprises. For more information, refer to Autoscaling Configuration.
High-Availability using Kubernetes: Infoworks support high-availability when the control plane is deployed on Kubernetes.
Map Roles and Domains to SAML groups:Infoworks now lets system administrators map SAML (e.g. Active Directory) groups to IWX roles and domains. Every time a new user is added, the integration with SAML manages user roles and domain accessibility. For more details, refer to Configuring roles and domains.
Parametrization: Infoworks provides the ability to pass run-time parameters to pipelines and other nodes in workflow via workflow parameters. For more information, refer to Workflow Parameters.
Ability to Exclude Columns during Ingestion (Column Projection): Infoworks now provides the ability to ingest a table with a selected subset of columns in both Datalake and CDW environments. You can now choose to exclude certain columns before the ingestion job is submitted. For more details, refer to Configuring a Table.
Mainframe Data Files Enhancements: Infoworks has now added the following features for Mainframe Data Files:
- Supports ingestion from Variable Block (VB) Mainframe Files.
- Provides the REST APIs to create and delete the file mappings for the Mainframe sources.
- Can ingest Copybook files which have incorrect indentation.
- Supports ingesting all types of mainframe files with filter.
- Supports ingestion Copybook files which have FILLER as a column name.
- Supports flattening of complex datatypes.
AVRO & ORC Source Connector (Native): Infoworks supports onboarding of ORC and AVRO files. This allows additional use cases for fast data migration. For more information, refer to Onboarding Data from AVRO and ORC.
New Cdata Source Connector Support : Infoworks now supports the following as Ingest sources - activecampaign, apachephoenix, btrieve, cockroachdb, databricks, googlespanner, graphql, informix, neo4j, sasdatasets, sasxpt, tableaucrm. For more information, refer to the list of CDATA connectors.
End-of-support for certain Source Connectors : Infoworks no longer supports the following connectors as source types - alfresco, awsdatamanagement, azuredatamanagement, datarobot, digitalocean, edgaronline, evernote, fedex, financialedgenxt, hpcc, openexchangerates, quandl, salesforcechatter, sapbusinessonedi, sfeinsteinanalytics, ups, usps, wasabi.
Enable Pre and Post jobs hooks in pipelines: Infoworks now provides the ability to add pre and post job hooks which can be run before or after a pipeline job in the data plane. For more information, refer to Job Hooks.
This section consists of the resolved issues in this release:
| JIRA ID | Issue |
|---|---|
| IPD-20889 | In a TPT-based teradata table, there is a discrepancy between the actual job run time and the job duration shown on the Job Metrics page. |
| IPD-20979 | The BigQuery pushdown attempts to read and write to the specified parent project instead of the configured project in the environment. |
| IPD-21093 | For a fixed length file, the first column gets moved to the last during ingestion. |
| IPD-21266 | The Pipeline lineage API does not return the source schema and source table name for pipelines built in Snowflake environment. |
| IPD-21308 | The Domain Summary page hangs when there are more than 100 pipelines in the domain. |
| IPD-21383 | Infoworks is overwriting spark driver and executor javaoptions resulting in users unable to add custom java options. |
| IPD-21503 | Operations Dashboard displays different count on speedometer chart and the actual list of jobs. |
The following section contains known issues that Infoworks is aware of, and is working on a fix in an upcoming release:
| JIRA ID | Issue |
|---|---|
| IPD-21496 | The Staging Dataset and Dataset fields are not getting populated correctly in the references table under BigQuery target node. In addition to this, Natural key is not appearing on the Properties tab in the pipeline. |
For Kubernetes-based installation, refer to Infoworks Installation on Google Kubernetes Engine (GKE).
For more information, contact support@infoworks.io.
To upgrade from 5.3.0.x to 5.4.1 Kubernetes, refer to Upgrading from 5.3.0.x to 5.4.1.