java.lang.NoClassDefFoundError commons exception IWException wile doing Parquet source Metacrawl

java.lang.NoClassDefFoundError: commons/exception/IWException wile doing Parquet source Metacrawl

Problem Description

Metadata Crawl for a Parquet Source fails with the below error in the job log.

Error: A JNI error has occurred, please check your installation and try again Exception in thread "main"java.lang.NoClassDefFoundError: io/infoworks/saas/ingestion/commons/exception/IWException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)

Root Cause

This issue happens if /opt/infoworks/lib/ingestion/commons.jar is missing in the ingestion_databricks_connect_classpath in conf.properties file.

Solution

a) Append /opt/infoworks/lib/ingestion/commons.jar to the beginning of ingestion_databricks_connect_classpath in $IW_HOME/conf/conf.properties file. This is under the assumption that /opt/infoworks is the home directory where Infoworks Data Foundry is installed.

b) cd $IW_HOME/bin

c) ./stop.sh ingestion-service

d) source env.sh

e) ./start.sh ingestion-service

f) Run the Parquet ingestion. This should resolve the issue.

Applicable Infoworks Data Foundry on Databricks Versions:

v3.2.x