Software Release Notes 5.4.2
Date of Release: May 2023
This section consists of the new features and enhancements introduced in this release. |
- SQL pipelines empower skilled data engineers to write free-form SQL for creating complex data pipelines. Data engineers can take advantage of this feature as an alternative to the no-code visual pipeline, enabling them to write custom SQL queries for data transformation and analysis. This new feature provides additional flexibility and control to users who want to customize their data pipelines. For more information, including limitations of this functionality, refer to Designing an SQL Pipeline.
- Infoworks can now validate columns during ingestion. Users can reject records if a field is null or if it does not match a regular expression pattern. For more information, refer to Data Validation.
- Infoworks now supports creating error tables and history tables in a staging schema or dataset instead of the target schema or dataset in the CDW environments. For more information, refer to Onboarding Data from RDBMS Source.
- Admins can view all the entities using the source extensions to understand the impact of any potential change. For more information, refer to Managing Generic Source Type.
- Infoworks now enables users to skip a specified number of trailer records when ingesting a file. For more information, refer to the "Number of Footer Rows" field in the Mapping File section.
- Users can apply SQL expressions during Ingestion. For more information, refer to Adding Transform Derivation.
- Users are now able to Ingest into existing tables in Infoworks Data Lake Data Environments. For more information, refer to Target Configurations.
- Users can configure target tables with CSV file format (with multi char and unicode delimiter support) for Infoworks Datalake Data Environments. For more information, refer to Delimited Text Files.
- You can use the Target Column Order feature to set the target table columns in a specific order while writing to ADLS as a data lake. For more information, refer to Target Column Order.
- Users are able to add a column to the target table during Ingestion. For more information, refer to Adding a Column to the Table.
This section consists of the resolved issues in this release:
| JIRA ID | Issue |
|---|---|
| IPD-21493 | The 5.4.0.x patches are merged to the current release. |
| IPD-21888 | You can now create pipelines via the API by using Environment Name, Environment Storage Name, and Environment Compute Template Name. |
| IPD-21449 | The Import SQL API is now picking the correct table (even if a table with the same schema/table name is present in multiple data environments). |
| IPD-21534 | The Initialize & Ingest and Truncate jobs can now reset the value of the last_merged_watermark key. |
| IPD-21792 | The duplicate tables are not allowed to be onboarded anymore on HIVE Metadata sync source. |
| IPD-21700 | Fixed the pipeline deletion issue. |
| IPD-20984 | Infoworks now supports creating error tables and history tables in staging schema/dataset instead of target schema/dataset in CDW environments. |
| IPD-21503 | Operations Dashboard now displays the same count on the speedometer chart and the actual list of jobs. |
| IPD-21451 | Pipeline Preview Data is working after connecting to a different port in Databricks Cluster. |
| IPD-21450 | Operations Dashboard now displays the Speedometer charts at the top of the page. |
| IPD-21729 | Password does not appear anymore in the logs when Snowflake source is being used. |
| IPD-21618 | The job_object.json file from Infoworks ingestion/pipeline job logs does not show Databricks token anymore (not even in the encrypted form). |
| IPD-21703 | The intermittent job failures due to token expiry are no longer happening. |
The following section contains known issues that Infoworks is aware of, and is working on a fix in an upcoming release:
| JIRA ID | Issue |
|---|---|
| IPD-22025 | If you set the Workaround: You must explicitly set the configuration with the intended value (true/false). |
| IPD-21928 | The user is unable to use Snowflake variables as Table Identifiers/Table Names. If used, it won't fetch/validate the metadata for the table. |
| IPD-21895 | Despite exclusion, the ziw_file_modified_timestamp column appears in the watermark column list. |
- You cannot use Snowflake variables as Table Identifiers/Table Names. If you try to use, it won't fetch/validate the metadata for the table.
- User-managed table options are not appearing for the Streaming sources in the CDW environment.
For Kubernetes-based installation, refer to Infoworks Installation on Azure Kubernetes Service (AKS).
For more information, contact support@infoworks.io.
To upgrade from 5.4.0.x to 5.4.2 Kubernetes, refer to Upgrading from 5.4.0.x to 5.4.2.
Was this page helpful?