v5.5.1
Date of Release: April 2024
This section consists of the new features and enhancements introduced in this release. |
- Airflow executor is changed to CeleryKubernetesExecutor to run deferrable tasks using celery worker and bash nodes using dedicated kubernetes pods. To know the details, refer to the notes section here.
- Infoworks now provides a log archival feature for Kubernetes-based installation, enabling periodic archiving and deletion of service logs and job logs to optimize disk usage. For more details, please refer to the section on Log Archival Configuration.
- Users can now export the data directly from Data Lake to SQL Server target table during Ingestion. Please refer to the Configuring SQL Server Target section here.
- Users can now optionally set origin 'from address' and the 'sender name' by setting the SMTP configurations - "email.fromAddress" and "email.fromName". Please refer here for details.
- Now Infoworks supports ADLS Gen2 for Teradata TPT Configurations. All the information on this can be found here.
- Support has been added for downloading dataplane logs for ingestion interactive jobs. You can view the details here.
- Users can now export the data directly from Data Lake to Oracle target table during Ingestion. Refer to the Configuring Oracle Target section.
- Now users can sort the snowflake target tables based on the column provided in the table configuration or pipeline target node configuration.
- Users can delete tables from a data source that are crawled but do not need to be ingested. Detailed information can be found in the auto$ page.
- Now the can handle the limitation set by databricks where job parameter length is greater than 10,000 characters. Refer to the note here.
- Users can now enable to configure the databricks cluster log path and job log path as per the user generalized path. Information on this can be viewed here.
This section consists of the resolved issues in this release:
| JIRA ID | Issue |
|---|---|
| IPD-25763 | Print Number of Records Inserted/deleted/updated to the snowflake pipeline/pipeline group logs. |
| IPD-25661 | Handling pipeline migration when reference table is not present to create a new table. |
| IPD-24093 | Config-maps are getting overwritten during upgrade |
| IPD-24529 | Subset of users unable to authenticate via SAML after upgrade to 5.5 |
| IPD-24201 | Unable to use derived column as watermark column for incremental Ingestion |
| IPD-24430 | Unable to delete or inactive the advanced key, if the key has trailing space |
| IPD-24258 | API Calls failed with "504 Gateway Time-out" Error |
| IPD-24502 | Support ingestion from Azure Event Hub using OAuth |
| IPD-24537 | Uncheck "enable watermark offset" is not working 5.5 |
| IPD-24546 | Ingestion job marked as failed after successfully ingesting data |
| IPD-24551 | Modify the manifest script to pick the correct manifest information. |
| IPD-24555 | Filter sources by associated_domains using Infoworks API in v5.5 |
| IPD-24598 | All the source tables(included in the domain) are not listed in the table mappings dropdown |
| IPD-24601 | All the reference tables are not listed in the reference table dropdown of the target configuration. |
| IPD-24602 | Pipeline config import not proceeding further, giving a blank screen. |
| IPD-24610 | Segmentation for Snowflake Tables - When using MOD in segmentation the query is failing in the background |
| IPD-24547 | Questions on Log Path location for Databricks clusters |
| IPD-24576 | Missing Teradata Drivers After Upgrade |
| IPD-24628 | Not able to delete Pipeline extension in v5.5 |
| IPD-24672 | Pipeline Config migration API automatically adds the audit columns to target node when the config json doesn't have any audit columns in target ( This is not acceptable for an existing target table) |
| IPD-24680 | Ingestion jobs failing with class not found exception after 5.5.0.2 upgrade |
| IPD-24696 | Change in behavior of upload schema in the 5.5 version |
| IPD-24740 | Infoworks Config Migration API fails to update table configurations for tables if any of the tables within the source have already been ingested. |
| IPD-24754 | Sql node pipeline failing with query validation |
| IPD-24759 | In User Patch endpoint, when username is updated associated domains gets removed |
| IPD-24747 | Update pipelines failing with invalid identifier of update column |
| IPD-24785 | Issue in Onboarding tables with same name from different schema. |
| IPD-24820 | Scheduled workflows continue to trigger even if we make the source unavailable to the domain |
| IPD-24821 | Enhancement request: Accessible sources and Make available in infoworks domains are not in sync |
| IPD-24825 | Pipeline Group Jobs Next URL in API is not working as expected. |
| IPD-24826 | Status of the pipeline is showing as Pending eventough pipeline-group job is aborted. |
| IPD-24833 | Unable to set schedule job time correctly when minute is chosen as '00' |
| IPD-24850 | RESTAPIs do not encode # in the url |
| IPD-24873 | Job Metrics are returned empty for a completed job via API |
| IPD-24939 | Configuration key to use custom port for Teradata TPT job |
| IPD-24944 | Persist airflow logs on volume mounts |
| IPD-24946 | Unable to add S3 storage in Databrics AWS. |
| IPD-25236 | Recrawl metadata throws "Provided Table Ids are not present in the Source" |
| IPD-25245 | Pipeline failed with SQL compilation error in v5.5.0.3 |
| IPD-25283 | set access-control/permission (access_control_list) when creating Databricks clusters is not working in v5.5.0.3 |
| IPD-25307 | User Managed tables should not be reference tables. |
| IPD-25341 | Bash nodes with custom image are failing on version 5.5.0.4 |
| IPD-25319 | Updating CSV Source is affecting stored access key details |
| IPD-25318 | Unable to save in UI while configuring same target details for user managed tables. |
| IPD-25372 | The source setup page refreshes continuously |
| IPD-25362 | Ingestion of snowflake table onboarded as a query as table failing with null pointer exception. |
| IPD-25448 | Questions on Log Path location for Databricks clusters |
| IPD-25368 | Add warnings on table's save action |
| IPD-25542 | Support Enhanced Flexibility Mode for Dataproc Clusters |
| IPD-25580 | Jobs failing randomly during cluster creation with File not found error after changing to dataproc 2.0 |
| IPD-25404 | Ingestion of snowflake table failing with Null pointer exception when we configure watermark offset in Ingest mode job run in v5.5.0.4 |
| IPD-25470 | PyCryptodome package version3.15.0 vulnerability |
| IPD-25479 | Enable watermark offset option gets disabled if we change the data type to string |
| IPD-25509 | Enhancement: Ability to receive the output of bash script to pipeline |
| IPD-25554 | Not able to override workflow variables in pipeline parameters |
The following section contains known issues that Infoworks is aware of, and is working on a fix in an upcoming release:
| JIRA ID | Issue |
|---|---|
| IPD-25836 | CDC pipeline builds with Insert Overwrite mode is failing on Azure databricks environment with DELTA storage format. |
| IPD-25881 | In snowflake on EMR environment, pipeline group jobs will fail when "run driver job on dataplane" is enabled. |
| IPD-25879 | Sync to target to oracle db fails for the latest oracle instance (Oracle Version 19) |
| IPD-25882 | NPE is noticed for Oracle sync to target when additional param field is empty. |
- Multiple pipelines executing in parallel on spark compute (dataplane) cause snowflake session failures leading to failed pipeline builds. Thus, It is recommended to run snowflake pushdown queries on compute plane.
For Kubernetes-based installation, refer to Infoworks Installation on Azure Kubernetes Service (AKS).
For VM-based installation, refer to VM-Based Deployment.
For more information, contact support@infoworks.io.
For upgrading from 5.5.0.x to 5.5.1 for Azure Kubernetes, refer to Upgrading Infoworks from 5.5.0.x to 5.5.1 for Azure Kubernetes.
For upgrading from 5.0 to 5.5.1 for VM, refer to Upgrading Infoworks from 5.5.0 to 5.5.1 for VM.
Was this page helpful?