dt_workspace_schema

Schema where Data Transformation stores all intermediate tables.

iw_dt_workspace

dt_workspace_base_path

HDFS path where Data Transformation stores data for all intermediate data.

/iw/df/workspace

dt_shared_connection_enabled

Whether to reuse the connection for interactive mode. Typically true for Hive+Tez and false for Hive+MR.

true

dt_shared_connection_timeout_ms

After how long should an idle shared connection be closed. Shared connection should be enabled.

180000

dt_interactive_exec_pool_size

Thread pool size for executing tasks per request in the interactive mode.

10

dt_interactive_table_maximum_size

Maximum size of any of the intermediate tables in interactive mode.

10000

dt_batch_exec_pool_size

Thread pool size for executing tasks per request in the batch mode.

5

dt_sampling_num_threads

Number of sampling tasks to run in parallel.

5

PIPELINE_SOURCES_AUTO_SYNC_CHECK_DISABLED

If set to true, it disables the error icon on the pipeline editor page indicating source schema mismatch. If set to false, the error icon displays on the pipeline editor page.

random_sampling_enabled

Determines whether sample generation, for source or pipeline target, needs to be random. To be disabled when sampling is taking too long to complete or is failing.

true

LOGMINER_TABLESPACE_NAME

Table space name.

use_new_tablespace

To start logminer in different tablespace.

False

ADVANCED_ANALYTICS_DISABLED

To disable advanced analytics nodes, this should be true.

False

modified_time_as_cksum

If this is true, the modified time is used to determine if the file has been changed or not. If it is set to false, the actual checksum is calculated.

false

delete_table_query_enabled

By default, Delete Query feature is available at the table level. Set IW Constant delete_table_query_enabled to false from UI to hide Delete Query feature.

true

Ufi_max_failure_percentage_per_table

The percentage of files for which the file ingestion is failed before the job is shown as failed.

0.0

MIN_ROWS_FOR_MERGEMR

Minimum number of rows in the current data in a secondary partition for the merge. If a secondary partition has number of rows less than this value, the merge jobs will be combined until they reach this threshold.

1000000

fail_ingestion_on_impala_cache_refresh_failure

Set to true if the ingestion job is to be failed in case Impala cache metadata is not refreshed.

iw_hive_ssl_enabled

To Enable or Disable SSL on Hive

false

iw_hive_ssl_truststore_path

Path to Hive Trust Store File Location

iw_hive_ssl_truststore_passwd

Encrypted Password for Trust Store

filesystem_scheme

The hdfs path will be pre-populated with the value set in this configuration. The values can be: s3, s3a, s3n, adls, wasb, gs.

dt_batch_sparkapp_settings

This configuration is set to overwrite spark configurations like executor memory, driver memory, etc during pipeline batch build.

dt_disable_sample_job

This configuration is set to disable sample jobs after pipeline build.

False

dt_disable_cache_job

This configuration is set to disable cache jobs after pipeline build.

False

CREATE_DROP_TARGET_SCHEMA

Used to allow Infoworks to run create database and drop database commands.If this value is set to false, the create database and drop database commands will not be executed.

True

target_schema_permission

Used to enable or disable creation of database in pipelines.If this value is set to false, creation of database in pipelines will be disabled.

True

number_of_parallel_jobs_per_entity

The number of jobs that can run in parallel for each entity.Entity refers to either of the following: Source, Datamodel, Cube.For example, if this value is set to 3, then a maximum of 3 source crawls or 3 data model builds or 3 cube builds can occur in parallel.

NA

dt_auto_exclusion_enabled

Used to set pipeline target column projection optimization on all nodes.

True

dt_merge_exec_pool_size

Number of concurrent tasks to run while performing merge on pipeline targets.

5

dt_fail_on_null_or_empty_partition_value

Used to fail the pipeline jobs when partition values are Null or Empty.

False

iw_udfs_dir

Indicates the UDFs directory.

NA

iw_hdfs_prefix

Default HDFS access protocol prefix.

hdfs://

dt_label_auto_cast_enabled

Used to enable Auto Cast mode for Advanced Analytics Model Label Column.

True

dt_label_cast_type

Used to indicate the default value for Advanced Analytics Model Label Column.

Double

dt_spark_master

Indicates the Spark master mode.

local

dt_target_hdfs_cleanup

To retain data on HDFS when the pipeline is removed, set this false.

True

dt_batch_spark_settings

Semi-colon separated list of Entity level Hive configurations. Applicable for batch mode pipeline build and to generate sample data in source.

key1=value1;key2=value2;key3=value3

dt_scd2_granularity

SCD2 record change granularity on timestamp can be Second,Minute,Hour,Day,Month,Year. This can be set using advance configuration to overwrite configuration for all targets at once.

second

dt_custom_udfs_force_copy

Custom UDFs in pipelines are only copied to HDFS when changes are detected. This configuration is used to overwrite changes to custom UDFs.

False

dt_disable_sample_job

Used to disable sample job for pipeline targets after pipeline build.

False

dt_disable_cache_job

Used to disable cache job for pipeline targets after pipeline build .

False

dt_spark_configfile

Spark configuration file path for Interactive mode Spark pipelines.

/etc/spark2/conf/spark-defaults.conf

dt_spark_configfile_batch

Spark configuration file path for Batch mode Spark pipelines.

/etc/spark2/conf/spark-defaults.conf

dt_batch_spark_coalesce_partitions

Spark coalesce configuration to create lesser files while writing to disk.

NA

dt_disable_current_loader

Custom Transformations cannot be loaded from current class loader. To load Custom Transformation from current classloader, set this configuration to False.

True

dt_overwrite_log_level

Log Level overwrite at pipeline level.For example, rootLogger=ERROR;io.infoworks=TRACE;infoworks=DEBUG;org.apache.spark=ERROR.

NA

dt_validation_progress_percent

Used to set the validation process percent in pipeline batch jobs.

10

dt_schemasync_progress_percent

Used to set the Schema Sync progress percent in pipeline batch jobs.

10

dt_spark_merge_file_num

Spark configuration to merge files using Coalesce option on dataframe during merge process.

1

iw_dt_ext_prefix

Pipeline extension prefix.

hdfs://

storage_format

Used to set the default storage format value for pipeline target.

ORC

user_extensions_base_path

Pipeline extension base path.

NA

SUPPORT_RESERVED_KEYWORDS

Used to enable or disable support for reserved keywords. To enables Hive connection on HDP 3.1, this value must be set to false.

True

dt_cutpoint_optimization_enabled

Enables caching when the value is set to true.

False

dt_redshift_s3_uri_scheme

This configuration is provided to specify the URI scheme for the S3 temp dir used in the Redshift connector.

s3a

dt_target_row_count_enabled

This configuration is to specify if the user wants to get the target row count at the end of pipeline build.

True

dt_label_auto_cast_enabled

This configuration is for auto casting labels in ML nodes for integers.

True

dt_label_cast_type

This configuration is to specify the cast type of the table column in ML nodes.

Double

dt_convert_source_watermark_to_timestamp

This configuration is for generic jdbc sources to convert source watermark column to timestamp.

False

dt_convert_ _all_source_watermark_to_timestamp

This configuration is for all sources to convert source watermark column to timestamp.

False

dt_source_type_sybase_iq_comparator_column

This configuration is for identifying the column to be used for fetching the latest records on the basis of timestamp for Sybase sources.

ZIW_OFFSET

dt_source_ type_oracle_comparator_column

this configuration is for identifying the column to be used for fetching the latest records on the basis of timestamp for Oracle sources.

ZIW_SCN

dt_source_type_sql_ server_comparator_column

This configuration is for identifying the column to be used for fetching the latest records on the basis of timestamp for Sql Server sources.

ZIW_SOURCE _START_TIMESTAMP

dt_source _type_db2_comparator_column

This configuration is for identifying the column to be used for fetching the latest records on the basis of timestamp for DB2 sources.

ZIW_SOURCE _START_TIMESTAMP

sample_data_row_count

Number of rows to be retrieved as part of sample data.

100

sample_data_error_threshold

% of invalid records in sample data after which the sample data will fail

50

file_preview_row_count

Number of rows to be shown for file preview.

100

file_preview_size_limit

Maximum File size for file preview.

10 (in MB)

bulk_payload_record_size

Number of records in the payload for each batch of bulk insert.

6500

maxNumOfRows

Maximum number of chunks allowed for segment load.

2000

error_record_threshold

Maximum number of invalid records after which the data crawl will fail.

1000

csv_null_value

Null value for csv file

Null

is_multi_line_csv

Set it to true if csv file has multilines.

False

CSV_FILE_ ENCLOSE_CHAR

Character used to enclose the column value.

"

CSV_FILE_ESCAPE_CHAR

The character used in the delimited files to escape the occurrences of column separator and column enclosed by in data.

\

exclude_partition_column_csv_export

If set to true, the output file will not contain the column on which file is partitioned

False

coalesce_partitions

Set the value to 1 if the output should contain a single file.

0

default_connectors_path

The base path for connectors.

lib/ingestion/connectors/

default_core_ingestion_lib_path

Path to the core libraries.

lib/ingestion/core/

use_iw_csv_mapper

false

tpt_files_base_path

Base path for all tpt job related file (configs,logs,scripts).

/temp/

add_tpt_ log_to_job_log

Add TPT log to job log.

True

copy_tpt_ log_to_target

Copy tpt log to target.

False

tpt_select_operator

Operator used for TPT export (EXPORT/SELECTOR).

EXPORT

tpt_charset

Charset specified in tbuild command.

ASCII

tpt_cloud_dont_split_rows

DontSplitRows parameter in tpt access modules.

True

use_tpt_generated_schema

Use TPT generated schema.

False

tpt_ascii_multiplication_factor

ASCII multiplication factor

1

tpt_utf8_multiplication_factor

UTF8 multiplication factor

2

tpt_utf16_multiplication_factor

UTF16 multiplication factor

2

tpt_export_operator_max_decimal_digits

MaxDecimalDigits attribute in select operator.

38

tpt_export_spool_mode

SpoolMode attribute in select operator.

spool

tpt_export_block_size

BlockSize attribute in select operator.

64330

tpt_export_tencity_hours

TenacityHours attribute in select operator.

4

tpt_export_retry_interval

TenacitySleep attribute in select operator.

1

tpt_export_max_sessions

MaxSessions attribute in select operator.

20

tpt_export_min_sessions

MinSessions attribute in select operator.

1

enable_tpt_data_encryption

DataEncryption attribute in select operator.

False

enable_tpt_trace_level

TraceLevel attribute in select operator (true will change it to ALL).

False

tpt_checkpoint_interval_in_seconds

-Z flag in tbuild command.

0

enable_tpt_log

-o flag in tbuild command.

False

tpt_cyclic_delivery

-C flag in tbuild command.

False

tpt_job_restart_limit

-R flag in tbuild command.

2

tpt_shared_ memory_size

-h flag in tbuild command.

10

merge_dedupe_function

De-duplication method used to de-duplicate cdc table (options : WINDOW, ORDER_BY_ DROP_DUPLICATES, SORT_DROP_DUPLICATES)

WINDOW

handle_deletes_during_merge

False

should_run_dedupe

De-duplicate the cdc data

False

should_run_de_dup_on_full_load

De-duplicate table during full load job (init and ingest).

False

get_sample_data_during_schema_crawl

Fetches sample data during metacrawl if set to true.

False

get_row_count_during_schema_crawl

Fetches row count during metacrawl if set to true.

False

metacrawl_batch_size

Number of tables to be metacrawled at a time

100

keywords_to_be_renamed_filepath

database_object_types_default

Table types to be included for browse source and meta crawl.

TABLE, VIEW, SYNONYM

keywords_to_be_renamed

USING

TEMP_LOG_TABLE_NAME

Name of the temporary log table created for tables configured for log based ingestion.

LOG_

NUMBER_OF_LOG_FILES_TO_LOAD

The configuration used to restrict the number of log files to load in each run of CDC.

-1

CHECK_FOR_NULL_LOG_FILE

The key that determines whether we check for null log file path in case of log based ingestion.

True

ORACLE_ARCHIVE_LOG_INFO_OBJECT_NAME

The name of the view of the oracle archive log info.

sys.v_$archived_ log

LOG_FILE_SEQUENCE_COLUMN_NAME

The column name of the column that maintains log file sequence.

SEQUENCE#

IS_SECONDERY_LOG_LOCATION

True if the user has changed the log file location, false otherwise.

False

include_current_log

The key that determines whether the key with the last log sequence number is included in the query or not.

False

ORACLE_LOG_ DESTINATION_ID

Destination id of the oracle log in case of secondary log location.

2

SET_NLS_DATE_FORMAT

The key that should be set to true if the user wants to use the NLS_DATE_ FORMAT.

False

NLS_DATE_FORMAT

This key specifies the default date format to use with the TO_CHAR and TO_ DATE functions.The value of this parameter can be any valid date format mask.

YYYY-MM-DD HH24:MI:SS

use_new_tablespace

To start logminer in different tablespace.

False

LOG_MINER_TABLESPACE_NAME

Tablespace name.

logmnrts

LOGMINER_TEMP_TABLE_TABLESPACE_NAME

Temp Tablespace name.

logmnrts

LOG_BASED_CDC_LOG_CONCAT_ORDER

ASC

ORACLE_LOG_BASED_CDC_CLEAN_CONTROL_CHARACTERS

False

WIDE_TABLE_ CONCAT_UDF_NAME

stragg(sql_redo_ record(sql_redo,query_order))

ADD_LOG_BASED_SELECT_QUERY_PARALLELISM

Parallelism factor for select queries for tables configured with log based ingestion. Should be an integer.

False

SELECT_QUERY_PARALLELISM_FACTOR

Parallelism factor for select queries. Should be an integer.

1

RUN_TEMP_TABLE_STATS

The key that determines whether we get the table stats for the temp table.

False

remove_extra_single_quote

The key used to remove any extra single quote char found.

True

escape_escape_char_in_data

The key that determines whether we use escape character while parsing the query.

True

date_format

Date format of data for log-based cdc.

dd-MMM-yy

timestamp_format

Time format of data for log-based cdc.

dd-MMM-yy hh.mm.ss a

is_time_format_needed

The key that determines whether we parse the date to a specific format while parsing the query.

True

use_jsql_and_expression

Should use jsqlparser AndExpression or not.

True

UNCOMMITED_TIME_PERIOD

The uncommitted records older than this time period (specified in days) are deleted.

7

oracle_timezone_as_region

If true, we set oracle.jdbc.timezoneAsRegion property to true. The property checks if the node connecting to oracle database is in some specified timezone list.

False

oracle_date_functions

Can specify any oracle date functions to handle date and time data easily and more effectively.

TO_DATE,TO_ TIMESTAMP

oracle_logbase_date_parsed_format

Oracle log based date parse format.

yyyy-MM-dd

oracle_logbase_timestamp_parsed_format

Oracle log based timestamp parsed format.

yyyy-MM-dd HH:mm:ss.SSSSSS

temp_table_stats_percent

Used to run temp table stats query.

20

temp_table_stats_degree

Used to run temp table stats query.

8

temp_table_ index_name_one

Prefix name used to create index on few columns of temp table.

IW_

CREATE_TEMP_TABLE_PARALLELISM

Parallelism factor for temp table queries. Should be an integer.

1

create_index_on_temp_table

An index is created on the temp table if set to true.

True

FILTER_ORACLE_COMMITTED_TRANSACTIONS

Option DBMS_LOGMNR.COMMITTED_ DATA_ONLY is set for the LogMiner session. This enables you to filter out rolled back transactions, transactions that are in progress, and internal operations.

False

ENABLE_DDL_TRACKING

Option DBMS_LOGMNR.DDL_ DICT_TRACKING is set for the LogMiner session to enable ddl tracking. DDL tracking enables LogMiner to successfully track structural changes made to a database object, such as adding or dropping columns from a table.

False

purge_retries_num_attempts

Number of retries if there is a failure during purge job.

5

should_send_notifications

The key that determines whether the user gets notified of events like, delete source, meta crawl, full load and cdc jobs, and so on.

True

governor_request_timeout_sec

Timeout for governor requests

30

custom_mappings

10000

oem_quote_character

Quote character to be used for oem source queries.

"

oem_key_default_value

Default key value used for oem connectors to connect.

XKR3Rwo8S5mLT57IGZ56uTIT/hcBbC1ZnYB+jYE1ZU02dCzvxakdhnuXFEY1I6HGmpiMg/AeHZJFuhb4w8bYkmQwFBkL8S+mM1bZva/zLbvxFG4kmRH4g1yFKQ==

meta_db_limit

It is required for fetching number of records in single call from metadb(mongo) if number of records are too large, and can’t be fetched in single call from metadb due to resources issue. Currently its being used while getting list of files in sources, if there files in order of 100k, we are fetching that detail in chunk, that limit describe that chunk size

1000

meta_crawl_row_count

100

topic_preview_row_count

Number of rows to be shown for topic preview.

100

kafka_erro_ threshold_default

This configures that the crawl fails if there are more error records than this configured limit, or more than successful records.

5

kafka_max_num_records_default

Maximum records to consider for schema crawl.

20000

kafka_streamin_frequency_num_default

Time unit for frequency type for schema crawl.

10

kafka_streamin_frequency_type_default

Streaming frequency for schema crawl.

Secs

kafka_streaming_mode_default

Streaming mode for schema crawl . Options are micro_batch and polling.

micro_batch

confluent_client_dns_lookup

Controls how the client uses DNS lookups.

use_all_ dns_ips

confluent_consumer_num_retries

This configures the Confluent consumer retries for metacrawl.

5

confluent_kafk_sample_data_error_threshold_default

Maximum number of invalid records after which the sample data will fail.

50

invalid_record_threshold

This configures that the crawl fails if there are more error records than this configured limit, or more than successful records.

5

confluent_deserializer_class

io.infoworks.connectors.streaming.confluent.kafka.utils.ConfluentJsonDeserializer - By default this class will be used to deserialize json messages which uses confluent KafkaJsonDeserializer.

cancel_job_via_file

If this is set to true the streaming job will be stopped by stopping the streaming query.

false true (for databricks)

logical_date_format

The format for specifying logical date for Avro messages.

YYYY-MM-dd

logical_time_millis_format

The format for specifying logical time in milliseconds for Avro messages

HH:mm:ss.SSS

logical_time_micros_format

The format for specifying logical time in microseconds for Avro messages

HH:mm:ss.SSSSSS

logical_timestamp_millis_format

The format for specifying logical timestamp in milliseconds for Avro messages

yyyy-MM-dd HH:mm:ss

logical_timestamp_micros_format

The format for specifying logical timestamp in microseconds for Avro messages

yyyy-MM-dd HH:mm:ss

local_timestamp_millis_format

The format for specifying local timestamp in milliseconds for Avro messages

yyyy-MM-dd HH:mm:ss

local_timestamp_micros_format

The format for specifying local timestamp in microseconds for Avro messages

yyyy-MM-dd HH:mm:ss

confluent_error_threshold_default

This configures that the schema crawl fails if there are more error records than this configured limit, or more than successful records.

5

confluent_max_num_records_default

Maximum records to consider for schema crawl.

20000

confluent_streaming_frequency_num_default

Time unit for frequency type for schema crawl.

10

confluent_streaming_frequency_type_default

Streaming frequency for schema crawl.

secs

confluent_streaming_mode_default

Streaming mode for schema crawl in confluent. Options are micro_batch and polling.

micro_batch

confluent_auto_offset

This configures the start point for a query. The "earliest" option ingests all the messages from the beginning, "latest" option ingests the messages that are pushed to topic after the query has started (which are the latest messages).

earliest

use_custom_deserializer

If set, the confluent_deserializer_ class value will be used for json message format. I the json messages are added to confluent topic using string serializer then set this configuration to false.

True

table_retrieval_mechanism

session_catalog

sfdc_socket_timeout_for_rest

Socket timeout when fetch mechanism is REST.

2000

sfdc_bulk_timeout

Timeout when fetch mechanism is bulk.

600000

sfdc_batch_size_for_rest

1200000

error_record_cache_count

100000

in_memory_cache_count

100000

serialize_hadoop_conf

True

sfdc_timeout

Connection timeout for salesforce soap api login.

1200000

ingestion_spark_configs

Null

ingestion_service_worker_pool_size

20

environment_home

/infoworks

run_optimize_table

Run OPTIMIZE table command for the table.

False

optimize_delta_write

This sets property for delta tables “delta.autoOptimize.optimizeWrite = true”.

False

auto_compact_delta_table

This sets property for delta tables “delta.autoOptimize.autoCompact = true”.

False

kafka_starting_offsets

This configures the start point for a query. The "earliest" option ingests all the messages from the beginning, "latest" option ingests the messages that are pushed to topic after the query has started (which are the latest messages).

Earliest

kafka_min_partitions

This configures the desired minimum number of partitions to read from Kafka. By default, Spark has a 1-1 mapping of topic partitions to Spark partitions consuming from Kafka.

Null

kafka_group_id

This configures the Kafka Group ID to be used in the Kafka consumer, while reading data from Kafka. By default, each query generates a UUID for reading data.

Null

kafka_optimize_query_batch_interval

This configures the number of batches after which the optimize query will be queried

10

kafka_consumer_num_retries

This configures the Kafka consumer retries for metacrawl.

5

kafka_meta_ crawl_error_threshold

This configures the threshold percentage for the meta crawl to fail. Meta crawl fails if the percentage of error records received is more than this configured value.

50

fail_on_data_loss

This configures whether the query must be failed when there is a possibility that the data is lost (For example, topics are deleted, or offsets are out of range).

False

kafka_extra_properties

This configures the start point for a query. The "earliest" option ingests all the messages from the beginning, "latest" option ingests the messages that are pushed to topic after the query has started (which are the latest messages).

Null

ui_url

The host for sending datacrawl notification.

Localhost

ui_port

The port for sending datacrawl notification.

3000

meteor_https_enabled

False

meteor_https_port

2000

rdbms_fetch_size

The JDBC fetch size, which determines how many rows to fetch per round trip

1000

mongo_date_time_format

DateTime format for mongo.

yyyy-MM-dd'T'HH:mm:ss.SSS'Z'

mongo_timezone

time zone for mongo

UTC

cdc_comparator_key

>=

tpt_segment_file_ts

000000000

file_split_size

134217728

data_types_ with_precision_scale

Datatypes which have precision and scale.

2,3

default_precision

Default precision of numeric and decimal datatypes.

38

default_scale

Default Scale of numeric and decimal datatypes.

18

use_query_for_meta_crawl_generic_jdbc

Whether to use query for metacrawl for generic jdbc sources.

True

stop_streamin_polling_frequency

15

use_type_cast

False

parquet_infer_schema

Whether to infer schema for parquet source.

False

delete_segment_staging_tpt

Whether to delete the staging area for teradata segments.

True

restricted_visibility_mode

When True, sources are private and accessible by other users only via domain based access. When False, only role based access is checked.

NOTE Platform restart is required when Infoworks REST APIs are in use and the value is changed.

False

iw_spark_app_conf

Spark configuration key to be used across ingestion and transformation

spark.key.1=value1 ; spark.key.2=value2 (values are separated by “;”)

export_metrics_capture

Configuration option to retrieve the export metrics. By default, this option is set to false. Set the value to true to retrieve metrics for export jobs.

False

tpt_file_escape_char

EscapeTextDelimiter attribute in TPT Data Connector operator

\

tpt_file_quote_escape_char

EscapeQuoteDelimiter attributte in TPT Data Connector operator

\

tpt_file_enclose_char

OpenQuoteMark and CloseQuoteMark attributes in TPT Data Connector operator

"

tpt_delimiter

TextDelimiter attribute in TPT Data Connector operator

|

tpt_io_buffer_size

IOBufferSize attribute in TPT Data Connector operator

1048575

tpt_use_hex_escape_chars

Use Hex escape character

False

tpt_cloud_prefix

Prefix base name for intermediate cloud in Data Connector initialization string

iw

tpt_cloud_single_part_file

SinglePartFile attribute in Data Connector initialization string

False

tpt_cloud_compress

Use compressed files for TPT dump

False

tpt_cloud_buffer_count

BufferCount attribute in Data Connector initialization string

Null

tpt_cloud_buffer_size

BufferSize attribute in Data Connector initialization string

Null

tpt_cloud_connection_count

ConnectionCount attribute in Data Connector initialization string

10

tpt_files_base_path

Base path for all tpt job related files (configs,logs,scripts)

/tmp/

tpt_cloud_config_directory

Relative path for TPT config files

tpt/cloudConfig

tpt_cloud_max_object_size

MaxObjectSize attribute in Data Connector initialization string

1024M

tpt_trace_level

TraceLevel attribute in Data Connector initialization string

Null

tpt_https_proxy

HttpsProxy attribute in GCP Access module initialization string

Null

tpt_add_script

Add TPT script to cluster logs

False

csv_comments_identifier

This is used to set a comment character in CSV files.

treat_empty_string_as_null

When set to false, empty string will be fetched as empty string, and not null

True

fetch_null_records

When set to true, records will be fetched with null watermark value

False

dt_interactive_service_cluster

You can choose whether to run the worker interactive service on control plane or data plane. Set the value to Databricks to run it on data plane.

Local

dt_snowflake_default_query_size_kb

Snowflake has a limitation of query size not being greater than ~520 KB. You can create cut points within the pipeline and break the query into smaller ones if the limit is being exceeded.

500 KB

dt_sf_cutpoint_optimization_enabled

Set this value to False to disable the cut point optimisation.

True

should_delete_tables_source_delete

When set to True, if the table is Infoworks managed, it will be deleted from snowflake database on deletion of the source. But if the table is user managed, it will just be truncated and not deleted from the Snowflake database.

NOTE This key can be set at each source level or at the global level.

False

dt_preview_data_disabled

When set to True, preview data is disabled globally and user gets a warning message "Preview data is disabled".

False

add_columns_for_alter_table

When set to True, the alter table queries runs to modify the exported table, or else user can alter the tables manually as per their need.

True

managed_mongo

When set to True, MongoDB is on Infoworks edge node, or else it is on MongoDb Atlas or external Mongo.

True

is_quiescent_enabled

When set to True, any new jobs that are submitted gets aborted. Any jobs that are already running will have no affect and continue till completion.

NOTE Users will not be able to start new jobs, but they can restart/resume the failed workflows.

False

system_name

This indicates the page title so that the user can correctly identify the IWX system/environment.

NULL

dt_bigquery_session_project_id

Use this configuration to specify the project id to use for the BigQuery session created for the BigQuery target jobs in dt.

<gcp_project_name>

ingestion_bigquery_session_project_id

Use this configuration to specify the project id to use for the BigQuery session created for the BigQuery target jobs in Ingestion.

<gcp_project_name>

enable_schema_sync

To enable schema evolution from source to Ingestion target table, set the configuration to true.

True