Databricks Cluster Launch Fails Due to Cluster Policy Mismatch

Databricks Cluster Launch Fails Due to Cluster Policy Mismatch

Issue Summary

When triggering an ingestion job in Infoworks using a Databricks Ephemeral Cluster, the cluster launch fails with a policy validation error. This is caused by a mismatch between the Spark extra Java options set by Infoworks and those enforced by the configured Databricks cluster policy.

Root Cause

Infoworks sends default Spark configuration during ephemeral cluster creation. However, if the configured cluster policy (e.g., C963C7783A0003B6) enforces a different set of values, the cluster validation fails with an error like:

pgsql

CopyEdit

INVALID_PARAMETER_VALUE : Cluster validation error: Validation failed for spark_conf,

spark.executor.extraJavaOptions must be -DIW_HOME=dbfs://infoworks_dit-dev-rel-a -Djava.security.properties= -Djava.net.useSystemProxies=true (is "-Djava.security.properties=");

Validation failed for spark_conf,

spark.driver.extraJavaOptions must be -DIW_HOME=dbfs://infoworks_dit-dev-rel-a -Djava.security.properties= -Djava.net.useSystemProxies=true (is "-Djava.security.properties=")

Default Spark Options Set by Infoworks

xxxxxxxxxx

text

CopyEdit

spark.executor.extraJavaOptions = -DIW_HOME=dbfs:// -Djava.security.properties=

spark.driver.extraJavaOptions   = -DIW_HOME=dbfs:// -Djava.security.properties=

However, the Databricks cluster policy expects additional flags such as:

xxxxxxxxxx

text

CopyEdit

-Djava.net.useSystemProxies=true

Resolution

To resolve this issue:

1. Update the Databricks cluster policy (ID: C963C7783A0003B6) to accept the Spark options provided by Infoworks or include the required parameters in the Infoworks configuration (via admin support).
2. Ensure the Spark options in the policy match exactly with what Infoworks sets during ephemeral cluster creation.
3. Re-run the ingestion job after updating the policy.

###