Infoworks Release Notes
Release Notes

v5.5.0.4

Date of Release: December 2023

Enhancement

JIRA IDIssue
IPD-24939*Configuration key to use custom port for Teradata TPT job.
IPD-24821*Accessible sources and make available in infoworks domains are not in sync.
IPD-24502

Support ingestion from Azure Event Hub using OAuth.

Limitation: Ingestion job for confluent (while using OAuth) should be submitted to a non interactive cluster.

Resolved Issues

NOTE The "*" symbol next to ID refers to the issues that have been resolved in the current release.

JIRA IDIssue
IPD-24747*Update pipelines failing with invalid identifier of update column.
IPD-24785*Issue in Onboarding tables with same name from different schema.
IPD-24820*Scheduled workflows continue to trigger even if we make the source unavailable to the domain.
IPD-24825*Pipeline Group Jobs Next URL in API is not working as expected.
IPD-24826*Status of the pipeline is showing as Pending even though the pipeline-group job is aborted.
IPD-24833*Unable to set schedule job time correctly when minute is chosen as '00'.
IPD-24850*REST APIs do not encode # in the URL.
IPD-24873*Job Metrics are returned empty for a completed job via API.
IPD-24754SQL node pipeline failing with query validation.
IPD-24740Infoworks Config Migration API fails to update table configurations for tables if any of the tables within the source have already been ingested.
IPD-24696Change in behaviour of upload schema in the 5.5 version.
IPD-24680Ingestion jobs failing with class not found exception after 5.5.0.2 upgrade.
IPD-24672Pipeline Config migration API automatically adds the audit columns to target node when the config json doesn't have any audit columns in target (This is not acceptable for an existing target table).
IPD-24576Missing Teradata Drivers after Upgrade.
IPD-24602Pipeline config import not proceeding further, giving a blank screen.
IPD-24601All the reference tables are not listed in the reference table dropdown of the target configuration.
IPD-24598All the source tables (included in the domain) are not listed in the table mappings dropdown.
IPD-24201Unable to use derived column as watermark column for incremental ingestion.
IPD-24529Subset of users unable to authenticate via SAML after upgrade to 5.5.
IPD-24537Uncheck "enable watermark offset" is not working 5.5.
IPD-24546Ingestion job marked as failed after successfully ingesting data.
IPD-24555Filter sources by associated_domains using Infoworks API in v5.5.
IPD-24529Subset of users unable to authenticate via SAML after upgrade to 5.5.
IPD-24093Config-maps are getting overwritten during upgrade.

Prerequisites

NOTE Before going through the below mentioned prerequisites for this section, ensure that all the Prerequisites for Installing Infoworks on AKS are validated.

  • Ensure the current deployment’s chart is present in /opt/infoworks.
  • Python 3.8 or later version with the pip module is installed in the Bastion VM.
  • Stable internet connectivity on the Bastion VM to download the required python packages from the python repository during installation/upgrades.
  • Validate that the version of the old chart is 5.5.0.3.
Command
Copy
Output
Copy

Ensure to take backup of MongoDB Atlas and PostgresDB PaaS. In case you don't take the backup, jobs will fail after the rollback operation. For more information, refer to the MongoDB Backup and PostgresDB PaaS Backup

Upgrade Instructions

NOTE Due to limitations in DataBricks interactive cluster session management, to ensure successful upgrade and usage of upgraded Infoworks libraries:

  1. Jobs using old versions of libraries must be completed/stopped.
  2. Old versions of libraries must be uninstalled and databricks cluster should be restarted to clear the cache.

For the procedure of libraries uninstalling, refer Step 4 below.

To upgrade Infoworks on Kubernetes:

It is assumed that the existing chart is placed in the /opt/infoworks directory and the user has the access permission.

Command
Copy

Before selecting the type of upgrade execute the following commands.

Step 1: Create the required directories and change the path to that directory.

Command
Copy

NOTE If your Infoworks installation is configured not to use the Infoworks hosted registry (IW_HOSTED_REGISTRY=false), you should download the Docker image ‘infoworks-mongo-utils_v5.5.0.tar.gz’ following the instructions that have been shared with you. After downloading, push this image to your configured registry, ensuring that you do not change the image tags.

Internet-free Upgrade

NOTE If you are upgrading via Internet-based procedure, skip to the next section.

Step 1: Download the upgrade tar files shared by the Infoworks team to the Bastion (Jump host) VM and place it under $IW_HOME/downloads.

Step 2: To configure Internet-free upgrade, execute the following command:

Command
Copy

Internet-based Upgrade

Step 1: Download the Update script tar file.

Command
Copy

Common Steps for Both Internet-free and Internet-based

NOTE Once you have selected the type of upgrade, the below mentioned steps are common for both Internet-free and Internet-based.

Step 1: Extract the iwx_updater_k8s_5.5.0.4.tar.gz under $IW_HOME/downloads.

Do not extract the tar file to /opt/infoworks/iw-k8s-installer as it would result in loss of data.

Bash
Copy

This should create two new files as follows - update-k8s.sh and configure.sh.

Step 2: Run the script.

Command
Copy

NOTE At the end of the above command's execution, if you want to run helm upgrade manually, then type N and press Enter. There is a 30-second timeout set to abandon the deployment of the upgraded version. If no input is received within the timeout duration, the deployment is triggered.

Output
Copy

Step 3: To modify any of the configurations listed below, follow the steps:

  1. Email configurations: smtpHost, smtpPort, smtpUsername, smtpPassword, sslEnabled
  2. Timeout for DT
  3. Timeout for ingestion
  4. nginx.ingress.kubernetes.io/proxy-body-size

Step 3A: Navigate to the directory IW_HOME/iw-k8s-installer. And Edit the values.yaml file

NameDescriptionDefault Values
Email configurationsmtpHostThe SMTP host URL to connect tosmtp.gmail.com
smtpPortSMTP port587
smtpUsernameThe SMTP User to authenticate asemail address
smtpPasswordThe Password for the SMTP userEncrypted password
sslEnabledThe SSL flagtrue
DTtimeoutSecondsTimeout for DT7200
IngestiontimeoutSecondsTimeout for Ingestion7200
nginx.ingresskubernetes.io/proxy-body-sizeNginx proxy body size10m

NOTE

  • Email configurations can be found under customiwConfigs section in values.yaml
  • Timeouts for DT and ingestion can be found under databricks section in values.yaml
  • kubernetes.io/proxy-body-size can be found in nginx.ingress section in values.yaml

Step 3B: After editing the annotations, the values.yaml file should look as shown below:

Update the values and save the file.

Step 3C: Navigate to the directory IW_HOME/iw-k8s-installer. And Run the iw_deploy script

Command
Copy

NOTE During the above command's execution, when it prompts if we need to override, type y and press Enter.

Step 3D: Restart all the deployments

Command
Copy

Step 4: (Applicable only for Databricks Persistent Clusters): A change in the Infoworks jar requires libraries being uninstalled and cluster restart. Without this step, there will be stale jars. Perform the following steps:

(i) Go to the Databricks workspace, navigate to the Compute page, and select the cluster that has stale jars.

(ii) In the Libraries tab, select all the Infoworks jars and click Uninstall.

(iii) From Infoworks UI or Databricks dashboard, select Restart Cluster.

Rollback

Prerequisites

  • Before executing the rollback script, ensure that IW_HOME variable is set.
  • Assuming Infoworks home directory is /opt/infoworks, run the below command to set the IW_HOME variable.
  • Validate that the version of the old chart is 5.5.0.4.
Command
Copy
  • Ensure the current deployment’s chart is present in /opt/infoworks.
  • Execute the below command to check the appVersion.
Command
Copy
Output
Copy

Ensure to restore MongoDB Atlas and PostgresDB PaaS. In case you don't take the backup, jobs will fail after the restore operation. For more information, refer to the MongoDB Restore and PostgresDB Restore.

Rollback Instructions

NOTE Since IW_HOME has been exported in the Prerequisites section mentioned above, the following steps can be executed from any location for users with read/write access to the aforementioned IW_HOME.

Step 1: Download the rollback script.

Command
Copy

Step 2: Place the Update script in the same directory as that of the existing iw-k8s-installer.

Step 3: Ensure you have permission to the $IW_HOME directory.

Step 4: Give executable permission to the rollback script using the below command.

Command
Copy

Step 5: Run the script.

Command
Copy

Step 6: You will receive the following prompt, "Enter N to skip running the above command to upgrade the helm deployment. (timeout: 30 seconds): ", type Y and press Enter.

NOTE If you want to run helm upgrade manually, then type N and press Enter.

Prompt
Copy

NOTE There is a 30-second timeout set to abandon the deployment of the downgraded version. If no input is received within the timeout duration, the deployment is triggered.