Pipeline build fails with Too many open files error

Pipeline build fails with "Too many open files" error

Problem statement: When user runs a considerable number (around 50) of pipeline builds with hive execution engine in parallel, the user may find that some pipeline builds fail randomly with the below error message

java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/conf.empty/mapred-site.xml (Too many open files)

Root cause and Solution: The hive jobs triggered by infoworks open conf files such mapred-site.xml, core-site.xml etc. When user runs lot of concurrent jobs, one can hit limit on number of open files that can be opened concurrently.

To resolve this issue, please increase the open files limit for infoworks-user, hive user and hadoop user by setting the below configurations in /etc/security/limits.conf in all master nodes.

xxxxxxxxxx

infoworks-user hard nofile 100000

infoworks-user soft nofile 100000

infoworks-user hard nproc 100000

infoworks-user soft nproc 100000

hive hard nofile 100000

hive soft nofile 100000

hive hard nproc 100000

hive soft nproc 100000

hadoop hard nofile 100000

hadoop soft nofile 100000

hadoop hard nproc 100000

hadoop soft nproc 100000