v5.4.1.13

Date of Release: February 2024

Enhancement

Resolved Issues

NOTE The "*" symbol next to ID refers to the issues that have been resolved in the current release.

Upgrade

Assuming IW_HOME variable is set to /opt/infoworks

Prerequisite

To support rollback after metadata migration, you need to take backup of metadata. Following are the steps:

Step 1: Install/Download MongoDB tool: mongodump. (if needed).

Step 2: Create a directory to store the database backup dump using the below command.

xxxxxxxxxx

mkdir -p $IW_HOME/mongo_bkp

cd $IW_HOME/mongo_bkp

Step 3: Use the below command to take a dump (backup) of the databases from the mongodb server.

If MongoDB is hosted on Atlas

xxxxxxxxxx

mongodump "mongodb+srv://<username>:<password>@<mongodb_server_hostname>/<db_name>"

If MongoDB is installed with Infoworks on the same VM

xxxxxxxxxx

mongodump "mongodb://infoworks:IN11**rk@localhost:27017/infoworks-new"

Procedure

For upgrading from 5.4.1/5.4.1.x to 5.4.1.13, execute the following commands:

Step 1: Use the deployer to upgrade from 5.4.1 to 5.4.1.13.

Step 2: Go to $IW_HOME/scripts folder of the machine.

Step 3: To ensure that there is no pre-existing update script, execute the following command:

[[ -f update_5.4.1.13.sh ]] && rm update_5.4.1.13.sh

Step 4: Download the update_5.4.1.13.sh

wget https://iw-saas-setup.s3.us-west-2.amazonaws.com/5.4/update_5.4.1.13.sh

Step 5: Give update.sh executable permission

chmod +x update_5.4.1.13.sh

Step 6 (Optional): If the patch requires Mongo Metadata to be migrated, run export METADB_MIGRATION=Y. This ensures that the metadata will be migrated, else run export METADB_MIGRATION=N.

Alternatively, you can enter it in the prompt while running the script.

Step 7: Update the package to the hotfix

source $IW_HOME/bin/env.sh

./update_5.4.1.13.sh -v 5.4.1.13-ubuntu2004

You will receive a "Please select whether metadb migration needs to be done([Y]/N)" message. If you need to perform metadb migration, enter Y, else, enter N.

Post Upgrade Steps

Steps to follow after upgrading Infoworks to 5.4.1.13:

NOTE Please make sure to take a backup of dataproc_defaults.json file before performing any change.

The dataproc_defaults.json file needs to be updated post upgrade to 5.4.1.13. This file is present in /opt/infoworks/conf directory. Here /opt/infoworks is the IW_HOME.

Following changes are required to be added in the dataproc_defaults.json file. To update the above file, change directory using cd /opt/infoworks/conf and run vi dataproc_defaults.json

Step 1: Add the property config.masterConfig.diskConfig.numLocalSsds : 0

Step 2: Add the property config.workerConfig.diskConfig.numLocalSsds : 0

Step 3: Add the object config.secondaryWorkerConfig :

xxxxxxxxxx

{

  "numInstances": 2,

  "instanceNames": [  ],

  "machineTypeUri": "https://www.googleapis.com/compute/v1/projects/#projectid/zones/us-central1-f/machineTypes/n1-standard-8",

  "diskConfig": {

    "bootDiskSizeGb": 50,

    "bootDiskType": "pd-standard"

  },

  "minCpuPlatform": "AUTOMATIC",

  "isPreemptible": true

}

Screenshot before update - No key with name secondaryWorkerConfig present inside config property.

Screenshot after update - Added the secondaryWorkerConfig object inside config property

Step 4: Add the array num_local_ssds : [0,1,2,3,4,5,6,7,8,16,24]

Screenshot before update - No field for num_local_ssds present

Screenshot after update - Added new Json array for num_local_ssds

Steps to Enable EFM on Dataproc

To add Secondary workers to a Dataproc cluster, select Enable Autoscale checkbox and then select the Enable Secondary Worker checkbox.

The Secondary worker type can be one of - spot VMs, standard preemptible VMs, or non-preemptible VMs.

As per the Dataproc documentation, following properties need to be added to enable EFM:

--properties=dataproc:efm.spark.shuffle=primary-worker \ --properties=dataproc:efm.mapreduce.shuffle=hcfs \

To add these properties, head over to the Advanced Configurations in the Compute section and and the following key:

Key: iw_environment_cluster_dataproc_config

Value: efm.spark.shuffle=primary-worker;efm.mapreduce.shuffle=hcfs

Additionally, the YARN graceful decommission time must be set to zero when EFM is enabled. To set that add the following advanced configuration:

Key: gracefulDecommissionTimeout

Value: 0 (zero)

Additional Notes

• The number of allowed local SSDs might differ based on the selected machine type. Please refer to - https://cloud.google.com/compute/docs/disks/local-ssd to check the allowed values.
• Clusters having local SSDs cannot be stopped.
• Cluster having secondary workers cannot be stopped. To stop, the secondary workers need to be scaled down to zero.
• Existing clusters cannot be updated from single-node to multi-node or vice-versa.
• When autoscale is enabled and advanced configurations for EFM are set, the Secondary workers must be enabled, else cluster creation will fail. This is because Primary workers cannot be autoscaled when Spark primary worker shuffle is enabled.

Rollback

Prerequisite

To rollback the migrated metadata:

Step 1: Install/Download MongoDB tool: mongorestore. (if needed)

Step 2: Switch to the directory where the backup is saved on the local system.

xxxxxxxxxx

cd ${IW_HOME}/mongo_bkp/dump

Step 3: Use the below command to restore the dump (backup) of the databases to the Mongodb Server.

If MongoDB is hosted on Atlas

xxxxxxxxxx

mongorestore "mongodb+srv://<username>:<password>@<mongodb_server_hostname>/<db_name>” --drop ./<db_name>

If MongoDB is installed with Infoworks on the same VM

xxxxxxxxxx

mongorestore "mongodb://infoworks:IN11**rk@localhost:27017/infoworks-new" --drop ./<db_name>

Procedure

To go back to previous checkpoint version:

Step 1: In a web browser, go to your Infoworks system, scroll-down to the bottom, and click the Infoworks icon.

Step 2: The Infoworks Manifest Information page opens in a new tab. Scroll down and check the Last Checkpoint Version.

Step 3: ssh to Infoworks VM and switch to {{IW_USER}}.

Step 4: Initialize the variables in the bash shell.

full_version=5.4.1.13

major_version=$(echo $full_version | cut -d "." -f 1-2)

previous_version=<Previous Version> # Last Checkpoint Version from step 1

os_suffix=<OS Suffix> # One of [ ubuntu2004 amazonlinux2 rhel8 ]

Step 5: Download the required deployer for the current applied patch.

https://iw-saas-setup.s3-us-west-2.amazonaws.com/${major_version}/deploy_${full_version}.tar.gz

Step 6: Execute the SCP command for the above mentioned files to the following path.

NOTE Remove the previously downloaded copy of deploy_${full_version}.tar.gz file in ${IW_HOME}/scripts/ directory.

${IW_HOME}/scripts/.

Step 7: Extract the deployed tar file in case it does not exist.

cd ${IW_HOME}/scripts

[[ -d iw-installer ]] && rm -rf iw-installer

tar xzf deploy_${full_version}.tar.gz

cd iw-installer

Step 8: Initialize the environment variables.

source ${IW_HOME}/bin/env.sh

export IW_PLATFORM=saas

Step 9: Run the Rollback command.

./rollback.sh -v ${previous_version}-${os_suffix}