Upgrading Infoworks from 6.1.2.x to 6.1.3 for Azure Kubernetes

Prerequisites

NOTE Before upgrade, user should cancel all their running jobs.

NOTE Before going through the below-mentioned prerequisites for this section, ensure that all the Prerequisites for Infoworks Installation on Azure Kubernetes Service (AKS) are validated.

NOTE Ensure that the helm version is 3.14 or above. This is mandatory from 5.5.1. You can install the latest helm client from here.

Follow the below steps to increase the limit of nginx proxy-body-size:

1. Navigate to the directory IW_HOME/iw-k8s-installer.
2. Edit the values.yaml file

xxxxxxxxxx

vi infoworks/values.yaml

a. Go to ingress.annotations section and edit the nginx.ingress.kubernetes.io/proxy-body-size to 200m.

b. After editing the annotations, the values.yaml file should look as shown below:

NOTE To check the supported version of MongoDB by Infoworks, refer to MongoDB Atlas Recommended Configuration.

• Ensure that the following Python packages are available on the server before starting the upgrade activity of Infoworks environment.

  • PyYAML (version 6.0)
  • jsonmerge (version 1.9.0)
  • wheel
  • pycryptodomex (version3.15.0)
  • argparse

• If the aforementioned packages are not available, execute the following command to install them.

xxxxxxxxxx

python3 -m pip install argparse pycryptodomex==3.15.0 PyYAML==6.0 jsonmerge==1.9.0 wheel

• Ensure the current deployment chart is present in /opt/infoworks.
• Python 3.8 or above version with the pip module is installed in the Bastion VM.
• Stable internet connectivity on the Bastion VM to download the required Python packages from the Python repository during installation/upgrades.
• Validate that the version of the old chart is 6.1.2.

xxxxxxxxxx

cat /opt/infoworks/iw-k8s-installer/infoworks/Chart.yaml | grep appVersion

xxxxxxxxxx

appVersion: 6.1.2.1

• User should enable quiescent mode config. Refer General Configuration.
• Cancel all the running jobs and workflow runs after waiting for some time.

NOTES

• User can view and cancel all running jobs from Manage Jobs in admin section.
• Users can view and cancel all workflow runs from Workflows in admin section.

Upgrade Instructions

To upgrade Infoworks on Kubernetes:

ASSUMPTION It is assumed that the existing chart is placed in the /opt/infoworks directory and the user has the access permission.

xxxxxxxxxx

export IW_HOME="/opt/infoworks"

Before selecting the type of upgrade execute the following commands:

Step 1: Create the required directories and change the path to that directory.

xxxxxxxxxx

mkdir -p $IW_HOME/downloads

cd $IW_HOME/downloads

Internet-free Upgrade

NOTE If you are upgrading via the Internet-based procedure, skip to the next section.

Step 1: Download the upgrade tar files shared by the Infoworks team to the Bastion (Jump host) VM and place it under $IW_HOME/downloads.

Step 2: To configure Internet-free upgrade, execute the following command:

xxxxxxxxxx

export INTERNET_FREE=true

Internet-based Upgrade

Step 1: Download the Update script tar file.

xxxxxxxxxx

wget https://iw-saas-setup.s3.us-west-2.amazonaws.com/6.1/iwx_updater_k8s_6.1.3.tar.gz

Common Steps for both Internet-free and Internet-based

NOTE Once you have selected the type of upgrade, the below-mentioned steps are common for both Internet-free and Internet-based.

Step 1: Extract the iwx_updater_k8s_6.1.3.tar.gz under $IW_HOME/downloads.

xxxxxxxxxx

tar xzvf  iwx_updater_k8s_6.1.3.tar.gz

This should create two new files as follows - update-k8s.sh and configure.sh.

NOTE Execute step 2 only if the "export KEYVAULT_ENABLED" flag in the $IW_HOME/iw-k8s-installer/configure.sh file is set to false and you want to enable Keyvault in the upgraded deployment, else skip to step 3.

Step 2: Edit the configure.sh file present in the $IW_HOME/downloads/ directory and fill details as given below. For more information related to Keyvault details, refer to Keyvault and Keyvault Authentication Configuration.

# Key Vault details

# Global KeyVault Flag for Infoworks

export KEYVAULT_GLOBAL_ENABLED="true"

​

# Keyvault enabled true for database details stored in keyvault

export KEYVAULT_ENABLED="true"

# if keyvault enabled is true, following parameters are required

export AZURE_KEYVAULT_URI="<Keyvault_URI>"

export FLAG_AZURE_KEYVAULT_AUTH_SP="true"

export AZURE_SERVICE_PRINCIPAL_TENANT_ID="<Tenant_ID>"

export AZURE_SERVICE_PRINCIPAL_SUBSCRIPTION_ID="<Subcription_ID>"

export AZURE_SERVICE_PRINCIPAL_CLIENT_ID="<Client_ID>"

export AZURE_SERVICE_PRINCIPAL_CLIENT_SECRET="<Client_Secret>"

export AZURE_MI_TYPE_IS_USER=""

export AZURE_USER_MI_CLIENT_ID=""

# export EXTERNAL_SECRET_STORE_TYPE=""

export KEYVAULT_FLAG_METADB_HOST="true"

export KEYVAULT_FLAG_METADB_USER="true"

export KEYVAULT_FLAG_POSTGRESDB_HOST="true"

export KEYVAULT_FLAG_POSTGRESDB_USER="true"

​

# External Databases Configurations, MongoDB Atlas

export MONGODB_HOSTNAME="METADBHOST"

export MONGODB_USERNAME="METADBUSER"

export MONGODB_ENCRYPTED_PASSWORD="METADBPASS"

​

# External Postgres Database configurations for PostgresDB on Cloud

export POSTGRESDB_HOSTNAME="POSTGRESDBHOST"

export POSTGRESDB_USERNAME="POSTGRESDBUSER"

export POSTGRESDB_ENCRYPTED_PASSWORD="POSTGRESDBPASS"

NOTE Execute step 3 only if the “export IW_LOG_ARCHIVAL” flag in the $IW_HOME/iw-k8s-installer/configure.sh file is set to false or not present and you want to enable Log Archival in the upgraded deployment, else skip to step 4.

Step 3: Edit the configure.sh file present in the $IW_HOME/downloads/ directory and fill details as given below. For more information related to Log Archival, refer to Log Archival Configuration.

xxxxxxxxxx

# Infoworks log archival

export IW_LOG_ARCHIVAL="true"

export IW_JOB_LOGS_MAX_AGE="<Number of days>"

Step 4: Run the script.

xxxxxxxxxx

./update-k8s.sh -v 6.1.3

NOTE At the end of the above command's execution, if you want to run helm upgrade manually, then type N and press Enter. There is a 30-second timeout set to abandon the deployment of the upgraded version. If no input is received within the timeout duration, the deployment is triggered.

xxxxxxxxxx

helm upgrade aks-upgrade-600 /opt/infoworks/iw-k8s-installer/infoworks -n aks-upgrade-600 -f /opt/infoworks/iw-k8s-installer/infoworks/values.yaml

Enter N to skip running the above command to upgrade the helm deployment. (timeout: 30 seconds): N

Upgrade success

Step 5: With 6.1.3, Infoworks recommends enabling network Policy for enhanced security. Please refer here for detailed documentation on how to enable network policies and for whitelisting additional namespaces. Run helm upgrade command again for the policies to take effect.

xxxxxxxxxx

helm upgrade aks-upgrade-610 /opt/infoworks/iw-k8s-installer/infoworks -n aks-upgrade-550_alpha -f /opt/infoworks/iw-k8s-installer/infoworks/values.yaml

Step 6: Run the following command to restart the deployment so that the Network Policy changes take effect.

xxxxxxxxxx

kubectl rollout restart deploy -n <namespace>

kubectl rollout restart sts -n <namespace>

Step 7: Disable quiescent mode:

(i) Infoworks will remain in quiescent mode which means the jobs will remain in pending state and the workflows will be paused.

(ii) User can provide test_workflow_ids and test_job_ids in the advanced configuration run the jobs and workflows to test the sanity of upgrade. Refer General Configuration.

NOTE Incase of any job failure, contact Infoworks support or move to rollback section directly.

To bring system out of quiescent mode user can run following commands:

xxxxxxxxxx

cd ${IW_HOME}/iw-k8s-installer/

xxxxxxxxxx

source functions.sh

xxxxxxxxxx

disable_quiescent

Step 8: (Applicable only for Databricks Persistent Clusters):

A change in the Infoworks jar requires libraries being uninstalled and cluster restart. Without this step, there will be stale jars. Perform the following steps:

(i) Go to the Databricks workspace, navigate to the Compute page, and select the cluster that has stale jars.

(ii) In the Libraries tab, select all the Infoworks jars and click Uninstall.

(iii) From Infoworks UI or Databricks dashboard, select Restart Cluster.

Rollback

Prerequisites

• Before executing the rollback script, ensure that IW_HOME variable is set.
• Assuming Infoworks home directory is /opt/infoworks, run the below command to set the IW_HOME variable.
• Validate that the version of the current chart is 6.1.3.

xxxxxxxxxx

export IW_HOME=/opt/infoworks

xxxxxxxxxx

appVersion: 6.1.3

Rollback Instructions

NOTE Since IW_HOME has been exported in the Prerequisites section mentioned above, the following steps can be executed from any location for users with read/write access to the aforementioned IW_HOME.

Step 1: Change into the Infoworks Charts Directory

xxxxxxxxxx

cd ${IW_HOME}

Step 2: Download the rollback script.

xxxxxxxxxx

wget https://iw-saas-setup.s3.us-west-2.amazonaws.com/6.1/rollback-k8s.sh

Step 3: Place the Update script in the same directory as that of the existing iw-k8s-installer.

Step 4: Ensure you have permission to the $IW_HOME directory.

Step 5: Give executable permission to the rollback script using the below command.

xxxxxxxxxx

chmod +x rollback-k8s.sh

Step 6: Run the script.

xxxxxxxxxx

./rollback-k8s.sh -v 6.1.2.1

Step 7: You will receive the following prompt. If you want to run helm upgrade manually, then type N and press Enter.

xxxxxxxxxx

"Enter N to skip running the above command to upgrade the helm deployment. (timeout: 30 seconds): ", type Y and press Enter.

NOTE There is a 30-second timeout set to abandon the deployment of the downgraded version. If no input is received within the timeout duration, the deployment is triggered.

Step 8: Disable quiescent mode:

(i) Infoworks will remain in quiescent mode which means the jobs will remain in pending state and the workflows will be paused.

To bring system out of quiescent mode user can follow these steps:

• Get encrypted METADB_PASSWORD from Chart configure.sh(unless stored on keyvault):

xxxxxxxxxx

cat ${IW_HOME}/iw-k8s-installer/configure.sh \

| grep -F "MONGODB_ENCRYPTED_PASSWORD=" | cut -d'=' -f2- | cut -d'"' -f2

• Get Orchestrator Pod Name:

xxxxxxxxxx

kubectl -n <namespace> get po | grep "orchestrator-[^scheduler|worker]"

• Exec into any orchestrator pod with the following commands:

xxxxxxxxxx

kubectl exec -it <orchestrator-pod> -n <namespace> -c orchestrator -- bash

• Inside the Pod, run:

xxxxxxxxxx

export METADB_PASSWORD=<MONGODB_ENCRYPTED_PASSWORD>

cd /opt/infoworks/logs

https://iw-saas-setup.s3.us-west-2.amazonaws.com/6.0/quiescent_mode_scripts.zip

unzip -o quiescent_mode_scripts.zip 2>&1 | grep -v -E "warning|Operation\ not\ permitted"

cd quiescent_mode_script 

bash iw_quiescent_mode.sh -e false