Kafka ingestion fails with error Caused by java.lang.NoSuchMethodError com.univocity.parsers.csv.CsvFormat.setDelimiter

Kafka ingestion fails with error Caused by: java.lang.NoSuchMethodError: com.univocity.parsers.csv.CsvFormat.setDelimiter

Problem Description:

Kafka streaming ingestion fails with below error

Caused by: java.lang.NoSuchMethodError: com.univocity.parsers.csv.CsvFormat.setDelimiter(Ljava/lang/String;)V

at infoworks.tools.filecrawler.generic.parser.csv.UnivocityCSVParser.(UnivocityCSVParser.java:54)

at infoworks.tools.filecrawler.generic.parser.csv.FlatFileParserFactory.getParser(FlatFileParserFactory.java:35)

at infoworks.tools.filecrawler.generic.parser.csv.FlatFileParserFactory.getParser(FlatFileParserFactory.java:141)

at consumers.spark.message.handlers.DelimitedMessageHandler.getValueMap(DelimitedMessageHandler.java:85)

at consumers.spark.message.mappers.DelimitedMessageMapper.parseRecords(DelimitedMessageMapper.java:195)

at consumers.spark.message.mappers.DelimitedMessageMapper.call(DelimitedMessageMapper.java:179)

Root Cause:

The issue happens when the univocity CSV parser is not able to parse the streaming data

Solution:

Configure streaming job to use COMMONS CSV parser instead of UNIVOCITY

Step 1: Make sure to have commons-csv-1.1.jar in the spark_streaming_job_classpath for the file $IW_HOME/conf/conf.properties

Step 2: Restart platform, restapi, queryservice.

cd $IW_HOME/bin

./stop.sh platform queryservice restapi

./start.sh platform queryservice restapi

Step 3: At the source configuration, add advanced configuration

CSV_PARSER_LIB = COMMONS

Rerun the streaming ingestion job again after making the above changes.

Applicable Infoworks DataFoundry Versions:

v.2.7, 2.8, 2.9, 3 x