Infoworks Installation on Google Kubernetes Engine (GKE)

NOTE If you have not installed Google Kubernetes Engine, refer to Installing Google Kubernetes Engine (GKE).

Prerequisites

NOTE

If you are using MAC OS to deploy Infoworks on to cluster, you must install the following package:

• Ensure that GKE Kubernetes cluster is connected to internet.

• Set up GKE Kubernetes cluster. For more information, refer to the Google Documentation.

• Ensure that Kubernetes version should be 1.25.x or above.

• Infoworks recommends creating the GKE Kubernetes cluster with private access and a VM as a Bastion host with Linux-based OS should be created within the VPC.

• If INGRESS_CONTROLLER_CLASS is set to nginx, then Infoworks recommends setting up ingress-controller externally with the required configuration. To set up nginx ingress-controller externally, refer to External Setup for Ingress Controller.

• If KEDA_ENABLED is set to true, then Infoworks recommends setting up KEDA externally with the required configuration. To set up KEDA externally, refer to External KEDA Setup.

• Install gcloud, Helm V3 and Kubectl on the Bastion host. Refer to official documentation for Helm and Kubectl installation instructions.

• To install pip module, run apt install python3-pip command.

• Ensure that the following Python packages are available on the server before starting the installation.

  • pycryptodomex (version3.15.0)
  • argparse

• If the aforementioned packages are not available, execute the following command

python3 -m pip install argparse pycryptodomex==3.15.0

• Verify the following prerequisites
  • Run gcloud version to ensure that gcloud is installed.
  • Run helm version to ensure that Helm is installed.
  • Run kubectl version to ensure that Kubectl is installed.
  • Run python3 -V to ensure that python (with venv) is installed.
  • Run sudo apt install python3-venv to install python Virtual Environment package.

• Set up Kubernetes Cluster in GKE for connection using gcloud by executing the following command:

NOTE The following procedure is a one-time activity for the user.

Step 1: gcloud auth login--launch-browser

After successful verification, the following confirmation message appears.

Step 2: Identify the cluster name, zone/region, and project you want to connect to. Run the following command with these details:

or

Persistent Storages

NOTE Google Kubernetes Engine RBAC Cluster Admin is required to run Infoworks installation.

Persistence ensures to persist the data even if a pod restarts or fails due to various reasons. Infoworks needs the following persistent storages to be configured:

• Databases (MongoDB and Postgres) and RabbitMQ
• Infoworks Job Logs and Uploads

Run the following command to fetch the storage classes:

NOTE The storageclass should have the reclaim policy Retain.

Filestore Mount Path

NOTE If you create Google Filestore manually, Infoworks configures the Filestore via NFS Client provisioner for the mounts in the Kubernetes Cluster from the Infoworks installation.

Step 1: Log in to Google Cloud console.

Step 2: Go to top-left of the navigation pane, under the Storage section, select Filestore > Instances.

Step 3: Click the Filestore instances you created. Once the server is in READY state. Get the details of the NFS mount point as shown below.

NOTE Keep a note of Filestore IP Address and Filestore Mount Path as they will be required during Infoworks installation.

Installing Infoworks on Kubernetes

NOTE Assuming IW_HOME=/opt/infoworks.

NOTE The following steps must be performed by the user who has completed the prerequisites procedure mentioned earlier.

Step 1: Create Infoworks directory under /opt.

sudo mkdir -p /opt/infoworks

Step 2: Change permissions of /opt/infoworks directory

sudo chown -R <user>:<group/user> /opt/infoworks

Step 3: Change the directory path to /opt/infoworks.

cd /opt/infoworks

Step 4: To download Infoworks Kubernetes template, execute the following command:

Step 5: Extract the downloaded file.

Step 6: Navigate to the extracted directory iw-k8s-installer.

Step 7: Open configure.sh file in the directory.

Step 8: Configure the following parameters as described in the table, and then save the file

NOTE Namespace and release names should not contain underscore (_). Release names should not start with numbers.

NOTE If you want to enable SSL before installing Infoworks, refer to Enabling SSL section.

Generic Configuration

Autoscaling Configuration

NOTE The total number of concurrent Infoworks jobs that can be submitted by each Hangman instance/pod is configured using num_executors in conf.properties. If the number of Hangman instances changes due to autoscaling, then the total number of jobs Infoworks handles also changes. To fix the total number of concurrent Infoworks jobs, you must disable the autoscaling on the Hangman service and set the number of Hangman replicas manually as described in the Enabling Scalability#enabling-scalability section.

External Container Registry Configuration

The following table lists the External Container Registry Configuration for Infoworks Setup. These configurations should be set only if the Container Registry used to pull the images is different from the one hosted by Infoworks.

The following fields are valid if IW_HOSTED_REGISTRY set to false

Use the Service Account key and the command listed below this table to create a secret in the Kubernetes namespace listed here.

For example:

Service Mesh Configuration for Security

NOTE Infoworks supports linkerd as of now.

NOTE Keep the following fields empty in configure.sh: KEYVAULT_GLOBAL_ENABLED, KEYVAULT_ENABLED, KEYVAULT_GLOBAL_ENABLED, KEYVAULT_ENABLED, AZURE_KEYVAULT_URI, FLAG_AZURE_KEYVAULT_AUTH_SP, AZURE_SERVICE_PRINCIPAL_TENANT_ID, AZURE_SERVICE_PRINCIPAL_SUBSCRIPTION_ID, AZURE_SERVICE_PRINCIPAL_CLIENT_ID, AZURE_SERVICE_PRINCIPAL_CLIENT_SECRET, AZURE_MI_TYPE_IS_USER, AZURE_USER_MI_CLIENT_ID, KEYVAULT_FLAG_METADB_HOST, KEYVAULT_FLAG_METADB_USER, KEYVAULT_FLAG_POSTGRESDB_HOST, KEYVAULT_FLAG_POSTGRESDB_USER

MongoDB Configuration

The following fields are applicable if EXTERNAL_MONGO= true.

PostgresDB Configuration

The following fields are applicable if EXTERNAL_POSTGRESDB= true

Filestore Configuration

Enable the following configuration to set up FileStore automatically using Infoworks installation.

NOTE It is assumed that gcloud is installed, configured, and user has Filestore edit permissions on GCP Cloud.

If FILESTORE_CREATION is set to true, then the below mentioned fields become valid.

If FILESTORE_CREATION is false and FILESTORE_AUTO_PROVISIONER is true.

Step 9 (Optional): Enable NodeSelector/Toleration and Custom annotations etc. by editing values.yaml file manually before deploying Infoworks deployment.

Step 10 (Optional): To run Infoworks jobs on separate workloads, edit values.yaml file under infoworks folder. Specifically, you need to edit jobnodeSelector and jobtolerations fields based on the node pool you created in the Node Pools

NOTE If you want to run Infoworks services on other node pools, you can edit nodeSelector and tolerations fields.

NOTE To run Infoworks Orchestrator workers on other node pools, you can edit workernodeSelector andworkertolerations fields.

Step 11 (Optional): To define the PaaS passwords, there are two methods:

First method

The password must be put in pre-existing secrets in the same namespace.

For MongoDB

(i) Set MONGODB_USE_SECRET_PASSWORD=true

(ii) To create the custom secret resource, run the following commands from the iw-k8s-installer directory.

NOTE Set the MONGODB_SECRET_NAME and IW_NAMESPACE according to the inputs given to the automated script. <mongo-password> is the plaintext password.

For Postgres

(i) Set POSTGRESDB_USE_SECRET_PASSWORD=true

(ii) To create the custom secret resource, run the following commands from the iw-k8s-installer directory.

NOTE Set the POSTGRESDB_SECRET_NAME and IW_NAMESPACE according to the inputs given to the automated script. postgres-password is the plaintext password.

Second Method

You can give the password to the Automated Script, which will encrypt it to store it in the templates.

Step 12: To run the script, you must provide execute permission beforehand by running the following command.

Step 13: Run the script

NOTE If you see this error, "INSTALLATION FAILED: failed post-install: timed out waiting for the condition", you can ignore this as it does not affect Infoworks installation.

Since the above installation was configured for ingress-controller, run the following command to get the domain mapping done.

NOTE Make sure to enable DNS mapping for IP address as per the above sample output.

Get the application URL by running these commands: http://sample.infoworks.technology.

Enabling SSL

If you set INGRESS_CONTROLLER_CLASS to nginx, add SSL Termination in the TLS section of values.yaml file either before running the automated script or after the deployment.

Step 1: Log in to Linux machine on the latest Debian-based OS.

Step 2: Ensure libssl-dev package is installed.

Step 3: Provide DNS Name for Infoworks deployment

Generating Self-Signed SSL Certificate:

To generate SSL, run the following commands:

NOTE Refer the following commands to replace "Infoworks.domain" and "subdomain.infoworks.domain" keywords with required domain and subdomain name.

Keep a note of server.crt and server.key files for self-signed certificates for Nginx SSL Termination and provide the valid values for ingress_tls_secret_name and namespace_of_infoworks.

Run the following command to add the tls certificates to the Kubernetes cluster.

Edit values.yaml file to look similar to the following sample file.

It is suggested to make changes in the values.yaml file and add the below parameters as annotations in the ingress block, replacing <URL> to the DNS of your deployment, as defined in IW_DNS_NAME.

After adding the annotations, the values.yaml file should look as shown below.

NOTE If you have already performed the deployment and edit the values.yaml file, then run helm upgrade command.

Custom Infoworks Configurations

Email Configuration

The following table lists the Custom Email Configuration for Infoworks Setup using custom email attributes.

Step 1: Update Email Configs as necessary.

• Navigate to the directory IW_HOME/iw-k8s-installer/infoworks.
• Edit the values.yaml file.
• Email configurations can be found under customiwConfigs section followed by emailSettings sub-section in values.yaml

Step 2: The structure and indentation of your values.yaml file should mirror the example provided below. Note that the actual values and entries may vary.

Update the values and save the file.

Step 3: Navigate to the directory IW_HOME/iw-k8s-installer. And Run the iw_deploy script.

NOTE During the above command's execution, when it prompts if we need to override, type y and press Enter.

Step 4: Restart all the deployments

Airflow Configurations

The following table lists the airflow configurations for Infoworks setup. Follow below instructions to update them as necessary.

Step 1: Update Airflow Configs as necessary.

• Navigate to the directory IW_HOME/iw-k8s-installer/infoworks.
• Edit the values.yaml file.
• Airflow configurations can be found under customiwConfigs section followed by airflowCfg sub-section in values.yaml

Step 2: The structure and indentation of your values.yaml file should mirror the example provided below. Note that the actual values and entries may vary.

Update the values and save the file.

Step 3: Navigate to the directory IW_HOME/iw-k8s-installer. And Run the iw_deploy script.

NOTE During the above command's execution, when it prompts if we need to override, type y and press Enter.

Step 4: Restart all the deployments

Dataproc Configurations

The following table lists the global Dataproc configurations for Infoworks setup. Follow below instructions to update them as necessary.

Step 1: Update Dataproc Configs as necessary.

• Navigate to the directory IW_HOME/iw-k8s-installer/infoworks.
• Edit the values.yaml file.
• Dataproc configurations can be found under gcpDataproc section in values.yaml

Step 2: The structure and indentation of your values.yaml file should mirror the example provided below. Note that the actual values and entries may vary.

Update the values and save the file.

Step 3: Navigate to the directory IW_HOME/iw-k8s-installer. And Run the iw_deploy script.

NOTE During the above command's execution, when it prompts if we need to override, type y and press Enter.

Step 4: Restart all the deployments

Enabling High-Availability and Scalability

Enabling High-Availability

Infoworks installation enables high-availability configuration while setting up Infoworks in Kubernetes. You can enable high-availability by editing the helm file called values.yaml.

Step 1: To edit values.yaml file, perform the action given in the following snippet.

Step 2: Run HELM upgrade command.

This enables the high availability for Infoworks.

Limitation For Kubernetes HA setup, local Postgres database is not completely HA compliant. In certain conditions, if the Postgres containers crash, it might result in workflow failures.

Enabling Scalability

Infoworks installation supports auto-scaling of pods.

For a scalable solution:

• There must be a minimum of two replicas, if HA is enabled.
• They can be scaled to any number based on available resources (CPU and memory).
• Infoworks supports scalability of source, pipeline, and workflow jobs out of the box. Ensure that there are available resources in the Kubernetes cluster.

Infoworks services will scale automatically based on the workloads and resource utilization for the running pods.

To modify any autoscaling configuration, edit the horizontalPodScaling sub-section under global section in the values.yaml file.

However, there are three pods which require manual scaling based on workload increase, namely platform-dispatcher, hangman, and orchestrator-scheduler.

There are two ways to enable scalability:

1. By editing the values.yaml file.

Step 1: Edit the values.yaml file.

NOTE The “deploymentname” mentioned in the above parenthesis is given just for the ease of understanding. This deployment name can be a platform-dispatcher, hangman, or orchestrator-scheduler with actual name.

For example:

Step 2: To increase the scalability manually, run HELM upgrade command:

2. Using Kubectl

NOTE The “deploymentname” mentioned in the above parenthesis is given just for the ease of understanding. This deployment name can be a “releasename-platform-id” with the actual name.

For example:

Optional Configuration

NOTE The following optional configurations hold true only when HA is enabled.

For setting up Pod Disruption Budget

A Pod Disruption Budget (PDB) defines the budget for voluntary disruption. In essence, a human operator is letting the cluster be aware of a minimum threshold in terms of available pods that the cluster needs to guarantee in order to ensure a baseline availability or performance. For more information, refer to the PDB documentation.

To set up PDB:

Step 1: Navigate to the directory IW_HOME/iw-k8s-installer .

Step 2: Edit the values.yaml file.

Step 3: Under the global section and pdb sub-section, set the enabled field to true.

Step 4: Run HELM upgrade command.

For setting up PodAntiAffinity

If the anti-affinity requirements specified by this field are not met at the scheduling time, the pod will not be scheduled onto the node. If the anti-affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. For more information, refer to the PodAntiAffinity documentation.

To set up PodAntiAffinity:

Step 1: Navigate to the directory IW_HOME/iw-k8s-installer .

Step 2: Edit the values.yaml file.

Step 3: Under the global section, set the podAntiAffinity field to true.

Step 4: Run HELM upgrade command.

LIMITATIONS If PodAntiAffinity is set to true, then node count = replicas+1. For example, let's assume that HA is enabled with two replicas, then we need to configure minimum three nodes running for the Infoworks Service Node Pool in GCP.

Increasing the Size of PVCs

To scale the size of PVCs attached to the pods:

Step 1: Note the storage class of the PVCs to be scaled.

Step 2: Ensure allowVolumeExpansion is set to true in the storageClass.

Step 3: Delete the managing statefulset without deleting the pods.

Step 4: For each PVC, upscale the size (ensure all PVCs attached managed by a single statefulset have the same size. For example, all Postgres managed PVCs must have the same size).

Step 5: Navigate to the helm chart used for Infoworks deployment.

Step 6: Edit the values.yaml file to update the size of the corresponding database to the new value.

Step 7: Run the helm upgrade command.

Warning

Above upgrade command will recreate all pods with the same PVCs.

Updating the MongoDB and PostgresDB Credentials

To update the MongoDB and/or PostgresDB credentials in the Infoworks deployment, follow the below given procedure.

Updating the MongoDB Credentials

Updating Encrypted Passwords Stored in values.yaml

There are two methods to update password:

Method 1

To update MongoDB encrypted passwords that are stored in values.yaml file, with the existing configure.sh file, use the IW_DEPLOY script to populate values.yaml:

Step 1: Download and untar the Infoworks kubernetes template, if not already present, according to the iwx-version in your existing deployment.

Step 2: If a new template was downloaded, replace the iw-k8s-installer/configure.sh as well as iw-k8s-installer/infoworks/values.yaml with the older file.

Step 3: Change the directory path to iw-k8s-installer.

Step 4: Replace the following values with a blank string in the configure.sh file.

Step 5: Run iw_deploy.sh. Once you receive "Seems like you have already configured Infoworks once. Do you want to override? y/n Default: n", enter “Y”. This will prompt the user to provide input for the values that were blank in the previous step. The script will then replace the infoworks/values.yaml file with the updated values.

Step 6: Run the following command to upgrade by specifying your namespace and helm release name according to the values given in the configure.sh file.

Method 2

To update MongoDB encrypted passwords, you can directly modify the values.yaml file.

Step 1: Download and untar the Infoworks Kubernetes Template, if not already present, according to the iwx-version in your existing deployment.

Step 2: If a new template was downloaded, replace the iw-k8s-installer/infoworks/values.yaml with the older file.

Step 3: Change the directory path to iw-k8s-installer directory.

Step 4: Generate the encrypted passwords as needed. To generate any encrypted string, execute the following command.

This generates your passwords in a secure encrypted format, which has to be provided in the following steps.

Step 5: Replace the following yaml keys with the new values in the infoworks/values.yaml file, if needed.

Step 6: Run the following command to upgrade by specifying your namespace and helm release name according to the installed kubernetes deployment specifications.

Updating Encrypted Passwords Stored as a Separate Secret

To update the MongoDB password:

Step 1: Run the following commands from the iw-k8s-installer directory.

Step 2: Restart all pods except the databases.

Updating the PostgresDB Credentials

Updating Encrypted Passwords Stored in values.yaml

There are two methods to update password:

Method 1

To update PostgresDB passwords that are stored in values.yaml file, with the existing configure.sh file, use the IW_DEPLOY script to populate values.yaml.

Step 1: Download and untar the Infoworks Kubernetes Template, if not already present, according to the iwx-version in your existing deployment.

Step 2: If a new template was downloaded, replace the iw-k8s-installer/configure.sh as well as iw-k8s-installer/infoworks/values.yaml with the older file.

Step 3: Change the directory path to iw-k8s-installer.

Step 4: Replace the following values with a blank string in the configure.sh file.

Step 5: Run iw_deploy.sh. Once you receive "Seems like you have already configured Infoworks once. Do you want to override? y/n Default: n", enter “Y”. This will prompt the user to provide input for the values that were blank in the previous step. The script will then replace the infoworks/values.yaml file with the updated values.

Step 6: Run the following command to upgrade by specifying your namespace and helm release name according to the values given in the configure.sh file.

Method 2

To update PostgresDB encrypted passwords, you can directly modify the values.yaml file.

Step 1: Download and untar the Infoworks Kubernetes Template, if not already present, according to the iwx-version in your existing deployment.

Step 2: If a new template was downloaded, replace the iw-k8s-installer/infoworks/values.yaml with the older file.

Step 3: Change the directory path to iw-k8s-installer.

Step 4: Generate the encrypted passwords as needed. To generate any encrypted string, execute the following command.

This generates your passwords in a secure encrypted format, which has to be provided in the following steps.

Step 5: Replace the following yaml keys with the new values in the infoworks/values.yaml file, if needed.

Step 6: Run the following command to upgrade by specifying your namespace and helm release name according to the installed kubernetes deployment specifications.

Updating Encrypted Passwords Stored as a Separate Secret

To update the PostgresDB password:

Step 1: Run the following commands from the iw-k8s-installer directory.

Step 2: Restart the orchestrator and orchestrator-scheduler pods.

Limitations

MongoDB Limitations

With HA enabled, scaling the pods from higher to lower has the following limitations:

• Pods need to be manually deleted from replication configuration.
• Disabling HA to Non-HA is not supported once HA is enabled.

Database Limitations

Applicable to PostgresDB, MongoDB, and RabbitMQ.

• PVC’s size can’t be decreased.
• Increasing a PVC’s size requires downtime.
• After downscaling pods, the extra PVCs needs to be manually deleted.

PostgresDB Limitations

In the current HA architecture, on Postgres connection disruption, airflow is unable to reconnect via new connection. Furthermore, the current Postgres proxy is too simplistic to handle connection pools. Hence, if a Postgres master goes down, all running workflows will fail.