v6.1.0
Date of Release: September 2024
This section consists of the new features and enhancements introduced in this release. |
- All postgres queries from orchestrator-worker pod will be retried for unexpected connection closed from server. This behaviour can be controlled from values.yaml file for K8S based deployments. In values.yaml file
customiwConfigs.airflow.schedulerDBRetryEnabled/workerDBRetryEnabled/triggererDBRetryEnabledcan be configured with true/false value to enable/disable postgres connection retries. - Introduced Locking/Unlocking of Tables capability for admins, to prevent accidental submission of jobs. You can click here to view detailed information.
- For Databricks Compute, users have the flexibility to specify where interactive queries will be executed during data transformations. This is controlled by the configuration parameter
dt_interactive_service_cluster. By default, it is set to local, meaning queries will run locally using Databricks Connect. However, when set to databricks (i.e.,dt_interactive_service_cluster = databricks), interactive queries will be executed through job submissions on Databricks clusters. Visit this section for details. - New CData OEM Connectors have been introduced, further diversifying the range of data sources offered by Infoworks.
- Configuration
retain_version_numberis added to retain the pipeline version number during the configuration migration of the pipeline. Details here. - > For Snowflake environments, a new feature is introduced with the ability to directly switch between SQL and visual pipelines by selecting the new pipeline version from any base version. Previously, this option was only available during the export/import process. Now, users can seamlessly toggle between SQL and the more intuitive visual pipelines without the additional steps of export/import pipeline. For more detailed information on how this feature works, please refer here.
This section consists of the resolved issues in this release:
| JIRA ID | Issue |
|---|---|
| IPD-26667 | helm upgrade failed with issue in running airflowdb-upgrade pod |
| IPD-26669 | Issue while saving Data env Mapping for Domains when access is managed through IDP |
| IPD-26673 | Remove read only root file system for bash node pods |
| IPD-26690 | Post 6.0 upgrade unable to add non .sh files in the jobhook |
| IPD-26702 | The status of the Streaming job is still running Under Job Status even the job was stopped/cancelled |
| IPD-26692 | Unable to migrate workflow from DEV to PROD from UI on 6.0 |
| IPD-26718 | Spark config spark.yarn.maxAppAttempts doesn't take effect if we set it as an advanced config in infoworks |
| IPD-26746 | Platform CPU Utilization hike to 2500+% |
| IPD-26749 | The pipeline build job is failing with Table not found error when the source table of the pipeline belongs to a different domain's pipeline (target table) |
| IPD-26745 | Issue with import source configuration |
| IPD-26794 | The pipeline version not getting mapped during workflow import |
| IPD-26870 | Ingestion to target table is failing with permission issues in Unity catalog |
| IPD-26881 | v6.0.0.2 - Make port number configurable via Helm for Azure Postgres (Orchestrator) |
- For a CSV source, CDC records will go to error table, if during CDC we add new columns and remove older columns in the same file.
- CDC Insert Overwrite ingestion jobs with derived split by columns with Query as a table is failing.
- Insert Overwrite ingestion jobs with 'Query as a tables' are failing when the column names have space in it.
- Ingestion for tables created using query as a table with incremental mode insert overwrite will fail when the column names have space in it for a CDC job.
- Ingestion for tables created using query as a table with incremental mode insert overwrite and derived splitby configured will fail for a CDC job.
- ERROR message is noticed while opening preview data in custom target node in a pipeline.
- Streaming jobs that have been stopped may show running state for the cluster job - users can verify that the job is actually stopped by observing that the number of batches run for that job does not increase after stopping it; more details here.
- Micro batches processing stop for streaming ingestion if the source stops streaming the data for Databricks runtime version 14.3.
- Pipeline build failing with NPE when read from merged/deduplicated table is selected.
- Pipeline node preview data request times out for initial few tries.
- Error message is noticed when attempting to delete ACT CRM source.
- When the table is ingested using Databricks 14.3 and in the pipeline when we check for preview data either with 11.3 or 14.3, API gets timed out.
For Kubernetes-based installation, refer to Infoworks Installation on Azure Kubernetes Service (AKS).
For more information, contact support@infoworks.io.
For upgrading from 5.5.1.x to 6.0.0 for Azure Kubernetes, refer to Upgrading Infoworks from 6.0.0.x to 6.1.0 for Azure Kubernetes.
Was this page helpful?