v5.4.1.2

Date of Release: May 2023

Resolved Issues

NOTE The "*" symbol next to ID refers to the issues that have been resolved in the current release.

JIRA ID

Issue

IPD-22721*

For Confluent Kafka source, the streaming_group_id_prefix configuration is now working as expected.

IPD-22615*

The Partition and the Clustering details are now appearing in the BigQuery table created via Infoworks pipeline.

IPD-22036*

Pipeline build now succeeds even when the target table for the BigQuery external target already exists and is clustered.

IPD-22090*

For the Delimited File target, the timestamp format can be configured as per the user requirements. This will be applicable for all the timestamp columns for that table.

timestampFormat=yyyy-MM-dd HH:mm:ss.SSS

timestampFormat=yyyy-MM-dd HH:mm:ss

IPD-22351*

Infoworks has added advanced configuration for setting BigQuery Session Project IDs for Data Transformation and Ingestion jobs (dt_bigquery_session_project_id/ingestion_bigquery_session_project_id)

IPD-21449

The Import SQL API is now picking the correct table (even if a table with the same schema/table name is present in multiple data environments).

IPD-21534

The Initialize & Ingest and Truncate jobs can now reset the value of the last_merged_watermark key.

IPD-21584

The Import SQL command is now able to fetch queries that contain backtick (`).

IPD-21700

Fixed the pipeline deletion issue.

IPD-21792

The duplicate tables are not allowed to be onboarded anymore on HIVE Metadata sync source.

Upgrade

Assuming IW_HOME variable is set to /opt/infoworks

Prerequisite

To support rollback after metadata migration, you need to take backup of metadata. Following are the steps:

Step 1: Install/Download MongoDB tool: mongodump. (if needed).

Step 2: Create a directory to store the database backup dump using the below command.

mkdir -p $IW_HOME/mongo_bkp cd $IW_HOME/mongo_bkp

Step 3: Use the below command to take a dump (backup) of the databases from the mongodb server.

If MongoDB is hosted on Atlas

mongodump "mongodb+srv://<username>:<password>@<mongodb_server_hostname>/<db_name>"

If MongoDB is installed with Infoworks on the same VM

mongodump "mongodb://infoworks:IN11**rk@localhost:27017/infoworks-new"

Procedure

For upgrading from 5.4.1/5.4.1.x to 5.4.1.2, execute the following commands:

Step 1: Use the deployer to upgrade from 5.4.1 to 5.4.1.2.

Step 2: Goto iw_home/scripts folder of the test machine.

Step 3: To ensure that there is no pre-existing update script, execute the following command:

[[ -f update_5.4.1.2.sh ]] && rm update_5.4.1.2.sh

Step 4: Download the update_5.4.1.2.sh

wget https://iw-saas-setup.s3.us-west-2.amazonaws.com/5.4/update_5.4.1.2.sh

Step 5: Give update.sh executable permission

chmod +x update_5.4.1.2.sh

Step 6 (Optional): This patch requires Mongo Metadata to be migrated. To export the environment variable, run export METADB_MIGRATION=Y. This ensures that the metadata will be migrated, else run export METADB_MIGRATION=N.

Alternatively, you can enter it in the prompt while running the script.

Step 7: Update the package to the hotfix source iw_home/bin/env.sh

./update_5.4.1.2.sh -v 5.4.1.2-ubuntu2004

You will receive a "Please select whether metadb migration needs to be done([Y]/N)" message. If you need to perform metadb migration, enter Y, else, enter N.

Rollback

Prerequisite

To rollback the migrated metadata:

Step 1: Install/Download MongoDB tool: mongorestore. (if needed)

Step 2: Switch to the directory where the backup is saved on the local system.

cd ${IW_HOME}/mongo_bkp/dump

Step 3: Use the below command to restore the dump (backup) of the databases to the Mongodb Server.

If MongoDB is hosted on Atlas

mongorestore "mongodb+srv://<username>:<password>@<mongodb_server_hostname>/<db_name>” --drop ./<db_name>

If MongoDB is installed with Infoworks on the same VM

mongorestore "mongodb://infoworks:IN11**rk@localhost:27017/infoworks-new" --drop ./<db_name>

Procedure

To go back to previous checkpoint version:

Step 1: In a web browser, go to your Infoworks system, scroll-down to the bottom, and click the Infoworks icon.


Step 2: The Infoworks Manifest Information page opens in a new tab. Scroll down and check the Last Checkpoint Version.

Step 3: ssh to Infoworks VM and switch to {{IW_USER}}.

Step 4: Initialize the variables in the bash shell.

full_version=5.4.1.2

major_version=$(echo $full_version | cut -d "." -f 1-2)

previous_version=<Previous Version> # Last Checkpoint Version from step 1

os_suffix=<OS Suffix> # One of [ ubuntu2004 amazonlinux2 rhel8 ]

Step 5: Download the required deployer for the current applied patch.

https://iw-saas-setup.s3-us-west-2.amazonaws.com/${major_version}/deploy_${full_version}.tar.gz

Step 6: Execute the SCP command for the above mentioned files to the following path.

NOTE Remove the previously downloaded copy of deploy_${full_version}.tar.gz file in ${IW_HOME}/scripts/ directory.

${IW_HOME}/scripts/.

Step 7: Extract the deployed tar file in case it does not exist.

cd ${IW_HOME}/scripts

[[ -d iw-installer ]] && rm -rf iw-installer

tar xzf deploy_${full_version}.tar.gz

cd iw-installer

Step 8: Initialize the environment variables.

source ${IW_HOME}/bin/env.sh

export IW_PLATFORM=saas

Step 9: Run the Rollback command.

./rollback.sh -v ${previous_version}-${os_suffix}