Upgrade Procedure

NOTE: Before you upgrade, read the Release Notes for the updates in the version.

Prerequisites

• Ensure that the Mongo DB is up and running.
• If Mongo HA is enabled, ensure that the Mongo DB is in primary or secondary state on all nodes. If the Mongo HA node is in unrecoverable state, perform the procedure mentioned in Resync Member of Replica Set.
• Cleanup the logs that are not required, using the following script. You can create a cron job to schedule this in your Infoworks environment.
• If Infoworks is configured with SSL-enabled LDAP Auth, ensure to add the LDAP server certificate to the Java Keystore on the edge node before performing the upgrade.

Upgrade Procedures

The upgrade logs will be available in <path_to_Infoworks_home>/platform/iw-installer-metadata/logs/upgrade.log.

CentOS/RHEL6

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

• Run the following command: source $IW_HOME/bin/env.sh
• Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
• Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
• Upgrade the Infoworks DF version using the following commands: ./update.sh -v <version-number>

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

• updates the metadata DB binaries
• sets up feature compatibility version

CentOS/RHEL7

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

• Run the following command: source $IW_HOME/bin/env.sh
• Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
• Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
• Upgrade the Infoworks DF version using the following commands: ./update.sh -v 2.9.0-rhel7

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

• updates the metadata DB binaries
• sets up feature compatibility version

AZURE

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

• Run the following command: source $IW_HOME/bin/env.sh
• Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
• Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
• Upgrade the Infoworks DF version using the following commands: ./update.sh -v 2.9.0-azure

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

• updates the metadata DB binaries
• sets up feature compatibility version

GCP

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

• Run the following command: source $IW_HOME/bin/env.sh
• Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
• Upgrade the Infoworks DF version using the following commands: ./update.sh 2.9.0-gcp

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

• updates the metadata DB binaries
• sets up feature compatibility version

EMR

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

• Run the following command: source $IW_HOME/bin/env.sh
• Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
• Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
• Upgrade the Infoworks DF version using the following commands: ./update.sh -v 2.9.0-emr

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

• updates the metadata DB binaries
• sets up feature compatibility version

Upgrade Considerations

• If metadata backup is not successful, the upgrade will be discontinued.
• If jobs are run during upgrade, the upgrade will be discontinued.
• If any step fails during upgrade, the Infoworks upgrade process will automatically rollback to the base version, without any inconsistency.
• The HDFS temp directory cache will be cleared after the upgrade.

NOTE: Rollback is automatically handled during the upgrade process.

Post Upgrade

After upgrading the Infoworks services, perform the following post-upgrade tasks:

• Verify if all the Infoworks services are running
• Obtain the Infoworks license key from Infoworks support (for upgrade from versions below 2.5.0)

Verifying Infoworks Services

Perform the following:

• Trigger a simple Ingestion, Data Transformation and Workflow job.
• Verify that the jobs are running correctly.

Obtaining Infoworks License Key

An admin can perform the following License Management operations from the Infoworks DF:

• Add license keys
• Update license keys
• View features added to the product

Adding License Key

To add a new license key follow the below steps:

• Login to the Infoworks DF.
• Navigate to Admin > License Manager. The Infoworks License page is displayed which includes the following sections:

Summary: Summary of the Infoworks License

• License Violation: The list of license compliance anomalies. For details see the License Violations section below.
• Installation Id: Unique installation ID for each server
• License Type: Type of license like dl_basic, dl_enterprise, etc.
• License Starts: Timestamp when the license was activated.
• License Ends: Timestamp when the license expires.

Features - Details of features added to the product

• License Feature: list of features.
• Value: value assigned to each feature. The features that are activated using the license key are displayed as Yes.

Install Your New Infoworks License: text box to paste the license key.

• Copy the Installation ID and mail it to Infoworks. The license key will be sent to you via mail.
• Copy the license key and paste in the Install Your New Infoworks License text box.
• Click Install New License. The license will be installed.

To update the license key, follow the same procedure. Adding a new license key will replace the existing license.

License Violations

Following is the list of license violations:

• Server not having a license.
• License being expired or not started.
• Registered users not being licensed.
• Usage of unlicensed source types.
• Usage of unlicensed feature (Data Transformation, Cube, etc).

License Warning Messages

The license compliance anomalies are displayed in the License Management page and warnings are displayed in the respective source, pipeline, workflow and cube pages.

• All the license violations are displayed in the License Management page.
• When you use an unlicensed source type, a warning is displayed for the particular Source Type in the Manage Source page.
• When you edit a pipeline without the required license, a warning is displayed in the pipeline editor page.
• When you edit a workflow without the required license, a warning is displayed in the workflow editor page.
• When you edit a cube without the required license, a warning is displayed in the cube editor page.

Post-upgrade Procedure

Infoworks DataFoundry now supports HDP 3.1, in addition to HDP 2.5.5 and HDP 2.6.4 versions. And, Python version has been upgraded to Python 3.6.9. This requires modifications in the $IW_HOME/conf/conf.properties .

The properties must be modified when upgrading Infoworks DataFoundry from previous versions to 2.8.0 version (irrespective of whether the HDP version is 3.1 distro or not).

NOTE: New installations of Infoworks DataFoundry 2.8.0 works automatically, without the modifications.

Environment: HDP

Change Request 1

• Navigate to the $IW_HOME/conf/conf.properties file.
• Remove $IW_HOME//lib/parquet-support/* from the iw_jobs_classpath key value.
• Ensure that the additional : at the end of the value is removed.

Change Request 2

• Navigate to the $IW_HOME/conf/conf.properties file.
• If the base version is below 2.8, replace the Hive client libraries (like /usr/hdp/current/hive-client/lib/…) in the iw_jobs_classpath key value with /usr/hdp/current/hive-client/lib/*.
• Ensure that the additional : at the end of the value is removed.

Environments: HDP, MapR, CDH, Azure, GCP, EMR

Change Request 3

• Navigate to the $IW_HOME/conf/conf.properties file.
• Remove $IW_HOME/lib/shared/* from the df_batch_classpath key value.
• Ensure that the additional : at the end of the value is removed.

Change Request 4

• Navigate to the $IW_HOME/conf/conf.properties file.
• Remove $IW_HOME/lib/shared/* from the df_tomcat_classpath key value.
• Ensure that the additional : at the end of the value is removed.
• Stop and start the transformation service using the following commands: source $IW_HOME/bin/env.sh; $IW_HOME/bin/stop.sh df; $IW_HOME/bin/start.sh df

Custom Jar Files

From Infoworks DataFoundry version 2.7.1, during an upgrade the $IW_HOME/lib/extras directory and the subdirectories are preserved to place the custom jars. Users having custom jars must manually place them in these folders.

The custom jar files or libraries related to ingestion, data transformation, cubes, export, Hive crawls, streaming, and cloud must be placed in the following folders respectively:

• Ingestion and Cube: $IW_HOME/lib/extras/ingestion
• Data Transformation: $IWHOME/lib/extras/dt
• Export: $IW_HOME/lib/extras/export
• Hive Crawl: $IW_HOME/lib/extras/hive-crawl
• Streaming: $IW_HOME/lib/extras/streaming
• Cloud: $IW_HOME/lib/extras/cloud

It is ensured that these folders are preserved during the Infoworks upgrade process. Any jar files placed outside these folders are deleted during the upgrade process.

Following are the folders to classpath key mapping:

• iw_jobs_classpath: $IW_HOME/lib/extras/ingestion
• df_batch_classpath, df_tomcat classpath: $IW_HOME/lib/extras/dt
• iw_export_classpath: $IW_HOME/lib/extras/export
• iw_hive_schema_crawl_classpath: $IW_HOME/lib/extras/hive-crawl
• spark_streaming_job_classpath: $IW_HOME/lib/extras/streaming
• iw_ecb_client_classpath: $IW_HOME/lib/extras/cloud

Post Infoworks DataFoundry 2.7.1 migration, these folders are placed at the beginning of their respective classpath keys.

Ensure to place all the custom jars in the respective folders. After Infoworks DataFoundry 2.7.1 upgrade, you must manually place all the custom jars added in different folders (in older versions) to the above mentioned folders, to ensure compatibility.