Ingestion is failing with io.infoworks.saas.ingestion.commons.exception.IWException io.infoworks.saas.ingestion.commons

Ingestion is failing with "io.infoworks.saas.ingestion.commons.exception.IWException: io.infoworks.saas.ingestion.commons.exception.IWException: Error while writing data" Error for csv source in iwx 4.4.0.1

Problem Description:

Ingestion is failing with "io.infoworks.saas.ingestion.commons.exception.IWException: io.infoworks.saas.ingestion.commons.exception.IWException: Error while writing data" Error for csv source(source file location is sftp) when the target is gs bucket in iwx 4.4.0.1. Sample stacktrace looks like below,

    
21/04/06 23:46:42 ERROR io.infoworks.connectors.files.csv.datacrawler.CSVDataCrawler: io.infoworks.saas.ingestion.commons.exception.IWException: io.infoworks.saas.ingestion.commons.exception.IWException: Error while writing data to gs://gcp-pres-demo-staging//iw/sources/member_360_org_b_gcp_schema/temp//opt/infoworks/sources/member_360/member360_org_b.csv    at io.infoworks.saas.ingestion.commons.sftpclient.SFTPClient.copyFilesMethod(SFTPClient.java:161)    at io.infoworks.saas.ingestion.core.connectors.file.datacrawler.AbstractFileDataCrawler.copyFilesFromSFTPServer(AbstractFileDataCrawler.java:178)    at io.infoworks.saas.ingestion.core.connectors.file.datacrawler.AbstractFileDataCrawler.getFileDetailsMap(AbstractFileDataCrawler.java:162)    at io.infoworks.connectors.files.csv.datacrawler.CSVDataCrawler.runJob(CSVDataCrawler.java:106)    at io.infoworks.saas.ingestion.core.datacrawler.Crawler.crawl(Crawler.java:58)    at io.infoworks.saas.ingestion.core.datacrawler.Crawler.runCrawlJob(Crawler.java:85)    at io.infoworks.saas.ingestion.core.datacrawler.DistJobsDriver.submitCrawlJob(DistJobsDriver.java:82)    at io.infoworks.saas.ingestion.core.datacrawler.DistJobsDriver.runJobs(DistJobsDriver.java:43)    at io.infoworks.saas.ingestion.core.datacrawler.DistJobsDriver.main(DistJobsDriver.java:26)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:498)    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: io.infoworks.saas.ingestion.commons.exception.IWException: Error while writing data to gs://gcp-pres-demo-staging//iw/sources/member_360_org_b_gcp_schema/temp//opt/infoworks/sources/member_360/member360_org_b.csv
Copy

**Root cause:**

This is a limitation in Infoworks which is fixed in future releases.

Solution:

This issue has been fixed in future releases and there is a patch available for IWX v4.4.0.1. Instructions are listed below.

Take the backup of core.jar and commons.jar present in /opt/infoworks/lib/ingestion/core/
Replace the core.jar and commons.jar present in /opt/infoworks/lib/ingestion/core/ with the jars attached.
Change the permission to 755 for these 2 jars.

chmod 755 core.jar

chmod 755 commons.jar

Rerun the ingestion job.

Applicable IWX versions:

IWX 4.4.0.1

Last updated on

Was this page helpful?