Infoworks Release Notes
Release Notes

v5.3.0.12

Date of Release: March 2023

Features and Enhancements

JIRA IDIssue
IPD-20373Infoworks provides the REST APIs to create and delete the file mappings for the Mainframe sources. The sample curl request can be found here
IPD-20276Infoworks now supports ingestion from Variable Block (VB) Mainframe Files. To add variable block mainframe file-mapping, you can select the same from the Record Type dropdown.
IPD-20175Infoworks can now ingest Copybook files which have incorrect indentation. You can change the value of comment_upto_char (Default Value: 6) and comment_after_char (Default Value: 6).
IPD-20173, IPD-20097, and IPD-20088Infoworks supports ingesting all types of mainframe files with filter.
IPD-20099Infoworks supports ingestion Copybook files which have FILLER as a column name.
IPD-20098Infoworks supports flattening of complex datatypes.
IPD-20685

Infoworks supports to register and de-register HIVE/Metastore UDFs. Following are the key-value pairs to register/de-register.

should_register_hive_udfs=true/false

should_deregister_hive_udfs=true/false

hive_udfs_to_register= <functionName1>:<fullyQualifiedclassNameImplementingFunction1>;<functionName2>:<fullyQualifiedclassNameImplementingFunction2>

hive_udfs_to_deregister: <functionName1>:<functionName2>

Resolved Issues

JIRA IDIssue
IPD-21423Pipeline build fails with "Unable to find table metadata for node" error message.
IPD-20206If user tries to stop the Orchestrator using stop.sh script, it stops all the components except the Orchestrator Engine Worker.
IPD-20921The backlog jobs gets stuck in pending state and do not progress resulting in other jobs also getting blocked.
IPD-21003During the pipeline build, Infoworks is unable to read the timestamp in the BigQuery output.
IPD-21093For a fixed length file, the first column gets moved to the last during ingestion.
IPD-20889In a TPT-based teradata table, there is a discrepancy between the actual job run time and the job duration shown on the Job Metrics page.
IPD-20979The BigQuery pushdown attempts to read and write to the specified parent project instead of the configured project in the environment.
IPD-20087The Clustering columns are missing in the Target BigQuery table.
IPD-20536Data Analyst and Data Modeller have permissions to preview the data in pipeline, view sample data, and generate sample data when configuring a table.
IPD-20022Despite disabling the dataset creation in the pipeline configuration, the pipeline still creates the schema.
IPD-19943There is no provision to configure the disk space for the Dataproc clusters.
IPD-19929In few scenarios, Upgrading from 5.3.0 to 5.3.0.5 crashes the Ingestion service.
IPD-19945Infoworks does not fetch the correct datatypes for the CDATA sources.
IPD-19821When service credential used in BigQuery target is different than service credentials used to create environment , sync to target fails with “Invalid JWT Signature” error.
IPD-19766For BigQuery export files, Sync to Target is failing when the table schema contains array type.
IPD-19853If number of characters in table name exceed 27 characters, then export to teradata is failing with "table_name_temp already exist" error.
IPD-19751Infoworks does not disable query caching while fetching schema from BigQuery.
IPD-19815There are incorrect log messages in Sync to Target for teradata job in 5.3.
IPD-19474The API POST call to Pipeline Config-Migration fails with generic error.
IPD-19545The list of data connections and GET data connections APIs are accessible only to Admin users.
IPD-19542When running ingestion on BigQuery environment, error table is not getting created on BigQuery dataset if the source has only one error record.
IPD-19339Despite cluster creation getting completed, the Creating Cluster timer duration keeps increasing.
IPD-19663The Workflows were failing due to the request timeout. Hence, they go directly to failed state without executing any of the tasks.
IPD-19701The API call to trigger Sync to Target for the table group configured with target data connection fails.
IPD-19753For JSON and streaming sources, if CDC data has column with empty values, it is marked as an error record.
IPD-20202If you manually change the datatype for a column after the metacrawl, Mainframe ingestion fails.
IPD-20174 and IPD-20172If a COBOL layout file does not have header field, Infoworks unable to crawl/ingest EBCDIC file.
IPD-20684Google has changed the return message for exception handling of autoscaling policies resulting in job failure.
IPD-20570The "Add tables to crawl" API is not working for BigQuery Sync source.
IPD-20455The dt advanced configurations to merge partitions is not taking effect in the pipeline job dt_batch_spark_coalesce_partitions.
IPD-20432The workloads were running in the Compute project rather than the Storage project where datasets are persisted.
IPD-20397Pipeline build fails when the source table column has trailing "%".
IPD-20207Data analyst and Modeler are unable to crawl metadata.
IPD-20371While configuring the BigQuery target, the columns are getting ordered alphabetically irrespective of the order user chooses.

Upgrade

Assuming IW_HOME variable is set to /opt/infoworks

Prerequisite

To support rollback after metadata migration, you need to take backup of metadata. Following are the steps:

Step 1: Install/Download MongoDB tool: mongodump. (if needed)

Step 2: Create a directory to store the database backup dump using the below command

Command
Copy

Step 3: Use the below command to take a dump (backup) of the databases from the mongodb server.

If MongoDB is hosted on Atlas

Command
Copy

If MongoDB is installed with Infoworks on the same VM

Command
Copy

Procedure

For upgrading from 5.3.0 to 5.3.0.12, execute the following commands:

Step 1: Use the deployer to upgrade from 5.3.0 to 5.3.0.12

Step 2: Goto iw_home/scripts folder of the test machine.

Step 3: Execute the following command:

rm update_5.3.0.12.sh

Step 4: Download the update_5.3.0.12.sh

wgethttps://iw-saas-setup.s3.us-west-2.amazonaws.com/5.3/update_5.3.0.12.sh.

Step 5: Give update.sh executable permission

chmod +x update_5.3.0.12.sh

Step 6 (Optional): This patch requires Mongo Metadata to be migrated. To export the environment variable, run export METADB_MIGRATION=Y. This ensures that the metadata will be migrated, else run export METADB_MIGRATION=N.

Alternatively, you can enter it in the prompt while running the script.

Step 7: Update the package to the hotfix source iw_home/bin/env.sh

./update_5.3.0.12.sh -v 5.3.0.12-ubuntu2004

You will receive a "Please select whether metadb migration needs to be done([Y]/N)" message. If you need to perform metadb migration, enter Y, else, enter N.

Rollback

Prerequisite

To rollback the migrated metadata:

Step 1: Install/Download MongoDB tool: mongorestore. (if needed)

Step 2: Switch to the directory where the backup is saved on the local system.

Command
Copy

Step 3: Use the below command to restore the dump (backup) of the databases to the Mongodb Server.

If MongoDB is hosted on Atlas

Command
Copy

If MongoDB is installed with Infoworks on the same VM

Command
Copy

Procedure

To go back to previous checkpoint version:

Step 1: In a web browser, go to your Infoworks system, scroll-down to the bottom, and click the Infoworks icon.

Step 2: The Infoworks Manifest Information page opens in a new tab. Scroll down and check the Last Checkpoint Version.

Step 3: ssh to Infoworks VM and switch to {{IW_USER}}.

Step 4: Initialize the variables in the bash shell.

full_version=5.3.0.12

unpatched_version=$(echo $full_version | cut -d "." -f 1-3)

major_version=$(echo $full_version | cut -d "." -f 1-2)

previous_version=<Previous Version> # Last Checkpoint Version from step 1

os_suffix=<OS Suffix> # One of [ ubuntu2004 amazonlinux2 rhel8 ]

Step 5: Download the required deployer for the current applied patch.

https://iw-saas-setup.s3-us-west-2.amazonaws.com/${major_version}/deploy_${unpatched_version}.tar.gz

Step 6: Execute the SCP command for the above mentioned files to the following path.

NOTE Remove the previously downloaded copy of deploy_${unpatched_version}.tar.gz file in ${IW_HOME}/scripts/ directory.

${IW_HOME}/scripts/.

Step 7: Extract the deployed tar file in case it does not exist.

cd ${IW_HOME}/scripts

[[ -d iw-installer ]] && rm -rf iw-installer

tar xzf deploy_${unpatched_version}.tar.gz

cd iw-installer

Step 8: Initialize the environment variables.

source ${IW_HOME}/bin/env.sh

export IW_PLATFORM=saas

Step 9: Run the Rollback command.

./rollback.sh -v ${previous_version}-${os_suffix}