Pipeline Executor Preconfiguration

Static configurations used for different execution engines are set in the dt_executor_configs.yaml configuration file located in the $IW_HOME/conf folder. This file can be edited for pipelines to work with different versions and configurations of Hive or Spark.

Sample dt_executor_configs.yaml File

    
xxxxxxxxxx
 
#Executor Configurations --- batchEngine: "SPARK" udfStatements: - "DROP FUNCTION IF EXISTS iwrowid" - "CREATE TEMPORARY FUNCTION iwrowid AS 'io.infoworks.awb.udfs.RowIdUDF'" - "DROP FUNCTION IF EXISTS iwltrim" - "CREATE TEMPORARY FUNCTION iwltrim AS 'io.infoworks.awb.udfs.LTrimUDF'" - "DROP FUNCTION IF EXISTS iwrtrim" - "CREATE TEMPORARY FUNCTION iwrtrim AS 'io.infoworks.awb.udfs.RTrimUDF'" - "DROP FUNCTION IF EXISTS iwtrim" - "CREATE TEMPORARY FUNCTION iwtrim AS 'io.infoworks.awb.udfs.TrimUDF'" - "DROP FUNCTION IF EXISTS iwuuid" - "CREATE TEMPORARY FUNCTION iwuuid AS 'io.infoworks.awb.udfs.UuidUDF'" - "DROP FUNCTION IF EXISTS iwsplit" - "CREATE TEMPORARY FUNCTION iwsplit AS 'io.infoworks.awb.udfs.SplitUDF'" - "DROP FUNCTION IF EXISTS iwdateloop" - "CREATE TEMPORARY FUNCTION iwdateloop AS 'io.infoworks.awb.udfs.LoopUDF'" - "DROP FUNCTION IF EXISTS iwgenrows" - "CREATE TEMPORARY FUNCTION iwgenrows AS 'io.infoworks.awb.udfs.GenRowsUDF'" - "DROP FUNCTION IF EXISTS iwtime_similarity" - "CREATE TEMPORARY FUNCTION iwtime_similarity AS 'io.infoworks.awb.udfs.TimeSimilarityUDF'" - "DROP FUNCTION IF EXISTS SeqId" - "CREATE TEMPORARY FUNCTION SeqId AS 'io.infoworks.awb.udfs.SeqIdUDF'" propertyStatements: partitionedTargetProperties: - "SET hive.exec.dynamic.partition = true" - "SET hive.exec.dynamic.partition.mode = nonstrict " - "SET hive.optimize.sort.dynamic.partition=true" --- batchEngine: "HIVE" udfStatements: - "ADD JAR ${iw_hdfs_udfs_path}/df-shared.jar" - "DROP FUNCTION IF EXISTS iwrowid" - "CREATE TEMPORARY FUNCTION iwrowid AS 'io.infoworks.awb.udfs.RowIdUDF'" - "DROP FUNCTION IF EXISTS iwltrim" - "CREATE TEMPORARY FUNCTION iwltrim AS 'io.infoworks.awb.udfs.LTrimUDF'" - "DROP FUNCTION IF EXISTS iwrtrim" - "CREATE TEMPORARY FUNCTION iwrtrim AS 'io.infoworks.awb.udfs.RTrimUDF'" - "DROP FUNCTION IF EXISTS iwtrim" - "CREATE TEMPORARY FUNCTION iwtrim AS 'io.infoworks.awb.udfs.TrimUDF'" - "DROP FUNCTION IF EXISTS iwuuid" - "CREATE TEMPORARY FUNCTION iwuuid AS 'io.infoworks.awb.udfs.UuidUDF'" - "DROP FUNCTION IF EXISTS iwsplit" - "CREATE TEMPORARY FUNCTION iwsplit AS 'io.infoworks.awb.udfs.SplitUDF'" - "DROP FUNCTION IF EXISTS iwdateloop" - "CREATE TEMPORARY FUNCTION iwdateloop AS 'io.infoworks.awb.udfs.LoopUDF'" - "DROP FUNCTION IF EXISTS iwgenrows" - "CREATE TEMPORARY FUNCTION iwgenrows AS 'io.infoworks.awb.udfs.GenRowsUDF'" - "DROP FUNCTION IF EXISTS iwtime_similarity" - "CREATE TEMPORARY FUNCTION iwtime_similarity AS 'io.infoworks.awb.udfs.TimeSimilarityUDF'" - "DROP FUNCTION IF EXISTS SeqId" - "CREATE TEMPORARY FUNCTION SeqId AS 'io.infoworks.awb.udfs.SeqIdUDF'" propertyStatements: commonProperties: - "SET hive.enforce.bucketing=true" - "SET hive.optimize.bucketingsorting=false" partitionedTargetProperties: - "SET hive.exec.dynamic.partition=true" - "SET hive.exec.dynamic.partition.mode=nonstrict" - "SET hive.optimize.sort.dynamic.partition=true"
Copy

For troubleshooting, see Troubleshooting YAML Executor Configuration.

Last updated on

Was this page helpful?