Pipeline build fails with Too many open files error

Pipeline build fails with "Too many open files" error

Problem statement: When user runs a considerable number (around 50) of pipeline builds with hive execution engine in parallel, the user may find that some pipeline builds fail randomly with the below error message

java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/conf.empty/mapred-site.xml (Too many open files)

Root cause and Solution: The hive jobs triggered by infoworks open conf files such mapred-site.xml, core-site.xml etc. When user runs lot of concurrent jobs, one can hit limit on number of open files that can be opened concurrently.

To resolve this issue, please increase the open files limit for infoworks-user, hive user and hadoop user by setting the below configurations in /etc/security/limits.conf in all master nodes.