v5.1.2
Date of Release: 13 January 2022
New Features and Enhancements
Component: Onboard Data
- Synchronizing Data From Source-To-Teradata Target: Infoworks now supports direct export of data from a source to a teradata target, without creating pipelines. For more details, see Configuring Teradata Target.
- Query based Ingestion: Infoworks now supports ingestion of data based on custom queries. Tables ingested using this method will be loaded fully in the first ingestion, and the subsequent ingestions will be incremental data ingestion. This feature supports multi table joins, joins across tables in different schemas, restatement, In line Functions and Stored Procedures in the select clause. For more details, see Onboarding an RDBMS Source.
- UDF Support in Source Extension: Infoworks now offers the ability to configure and use the existing Hive UDFs. For more details, see Managing Source Extensions.
- Restatement Window Support for Incremental Ingestion: Infoworks now allows the restatement of incremental data for all native RDBMS connectors. This option is applicable for Timestamp and Date watermark columns which enables an additional offset (decrement) to the starting point for ingested data. Records created or modified within the offset time period are included in the next incremental ingestion job. For more details, see Configure Synchronization.
- Configurable Comment Character in CSV Ingestion: Infoworks now allows configurable comment character in CSV ingestion. For more details, see General Configuration.
- REST API added for Interactive Jobs Status: Infoworks now allows to get the status of interactive jobs using REST API, to understand if the interactive job has succeeded or failed. For more details, see Get Interactive Jobs List.
- Additional Filters added to REST API to Browse Source Tables: Infoworks now enables to browse only selective source tables based on schema or database or table name using REST API. For more details, see Browse source.
- Column Level Lineage Accessible using REST API: Infoworks now allows to access the column level lineage for pipeline using REST API. For more details, see Gets lineage for Pipeline Column.
- CDATA Drivers Updated to the Latest Version: Infoworks has now upgraded the CDATA drivers to its latest version (21.0.8011.0).
Known Issues
| JIRA ID | Issue Description |
|---|---|
| IPD-16376 | The table stored in Read Optimised (ORC) format can only be queried using spark, and not hive. |
| IPD-16499 | The workflow_id property is incorrectly named as parent_id for jobs submitted using a workflow. The workflow_id is incorrectly set to NULL in job metrics and custom audit columns for these jobs. |
| IPD-16504 | The Workflow Runs tab does not load the tables when there is unexpected metadata. |
Resolved Issues
| JIRA ID | Issue | Severity |
|---|---|---|
| IPD-16078 | The API for export job metrics is not returning the workflow id and workflow run id in API response | Highest |
| IPD-16289 | The APIs for Pipeline config migration are failing when there are advanced configs in the pipelines | Highest |
| IPD-16131 | Details missing for pipeline config migration v3 api in the Infoworks documentation | Medium |
| IPD-16227 | Hive source metacrawl is showing int family datatypes as a string in the UI/metadata | High |
| IPD-16069 | Username, passwords and related keywords to be removed from the log files | Highest |
| IPD-16226 | Upgrade NginX to the latest available version | High |
| IPD-16305 | Updating the max worker node count is taking no effect in dataproc cluster | Highest |
Installation
For the installation procedures, see Infoworks Installation on Azure, Infoworks Installation on AWS, Infoworks Installation on GCP.
Upgrade
For upgrading from lower versions to 5.1.2, see Upgrading to 5.1.2.
PAM
For the PAM, see Product Availability Matrix.
Was this page helpful?