System Configuration

The System Configuration page allows you to add/delete configurations and change the values and descriptions of a configuration key.

The autocomplete feature allows you to select the required configuration from the drop-down list based on your input.

Click the Admin menu and click the Configuration icon. The System Configuration page is displayed.


The following table lists all the configuration parameters and their descriptions.

System Configuration Keys and Descriptions

Configuration Parameter

Description

Default Value

hiveConfigurationVariables

Extra parameters for the hive connection. Default when we ship must be 'hive.auto.convert.join=false; hive.insert.into.multilevel.dirs=true'

'hive.auto.convert.join=falsehive.insert.into.multilevel.dirs=true' NOTE: Other values can be:hive.auto.convert.join=false;hive.insert.into.multilevel.dirs=true;hive.mapred.supports.subdirectories=true; mapred.input.dir.recursive=true; hive.optimize.insert.dest.volume=true; hive.exec.parallel=true;

numRowsSampleData

Number of rows to be retrieved as part of sample data.

100

numGrpsRange

Number of groups to be created while computing categorical data.


dw_join_numreducers

Number of reducers for data warehouse join.

10

dw_join_parallelism

Number of parallel joins for data warehouse build.

3

disable_categorical_data

If true, categorical data computation will be skipped.

false

source_ingest_parallelism

Number of tables to crawl in parallel.

2

disable_additional_table_info

If true, fetching sample data and computing categorical data will be skipped.

false

source_ingest_mappers

Number of mappers to crawl a table.

1

sqoopNumberOfReducers

Number of reducers to crawl a table

20

number_of_secondary_partition

Number of secondary partitions in each primary partition

2

extract_row_limit

Number of rows to crawl from a table per mapper, -1 will do full table crawl without any limit.

0

END_DATE_STRING

Represents expiration date for each record on Hive. This is an IW audit column.

9999-09-09

END_TIMESTAMP_STRING

Represents expiration timestamp for each record on Hive. This is an IW audit column.

9999-09-09 09:09:09

disable_row_count

If true, metadata crawl will not fetch row count for source tables.

false

iw_job_timeout_secs

Job timeout in milliseconds for Infoworks MR jobs.


iw_jobs_default_mr_map_mem_mb

Default map memory for Infoworks jobs.


iw_jobs_default_mr_red_mem_mb

Default reducer memory for Infoworks jobs.


fetch_size

In RDBMS ingestion using JDBC, this parameter can be added as a constant at the table level. This takes an integer (default size: 5000). This parameter is used to fetch at once, the number of rows using JDBC, which improves the performance for large table ingestions.

5000

cube_star_schema_job_map_mem_mb

Cube starschema job mapper memory.


cube_star_schema_job_red_mem_mb

Cube starschema job reducer memory.


max_number_of_chunks

Maximum number of chunks allowed while chunking a table.

2000

SKIP_RELOADED_CHUNKS

If true, already loaded chunks will be skipped during chunk loading.

true

enable_schema_synchronization

If true, table schema synchronization will be triggered before data synchronization.

false

db_time_zone

Database server time zone.

MST

NUM_PARALLEL_MERGE_TASKS

Number of parallel merge tasks while merging a table.

1

MERGE_CACHE_MB

Cache size for merging delta files.

0

source_crawl_cdc_mapmerge_mr_map_mem_mb



check_number_of_partitions

If false, checking available number of partitions during full load of a table will be disabled.

false

NUM_PARALLEL_MERGE_INMEM_LOADS

Number of parallel merge jobs that will be loaded in memory.

1

USE_COMBINE_INPUT_FORMAT

true


source_crawl_cdc_mapmerge_mr_red_mem_mb



parallel_table_merge

Number of different tables merge that can happen in parallel.

5

num_parallel_jobs_per_entity

Number of jobs that can run in parallel per entity (source crawl, data warehouse build, cube build).

2

disable_row_count_for_hive


false

recrawl_table_schema_on_table_truncate

If set to true, on table truncate, table schema will be re-crawled to get latest schema from source.

true

validate_chunks

If set to true, after every chunk load, data validation at chunk level will be triggered.

true

enable_job_logs_to_mongo

If set to true, job logs will go to file and mongo.

false

hive_conn_string_delimiter

Delimiter to use while forming Hive JDBC connection string.

?

df_workspace_schema

Schema where Data Transformation stores all intermediate tables.

iw_df_workspace

df_workspace_base_path

HDFS path where Data Transformation stores data for all intermediate data.

/iw/df/workspace

df_shared_connection_enabled

Whether to reuse the connection for interactive mode. Typically true for Hive+Tez and false for Hive+MR.

true

df_shared_connection_timeout_ms

After how long should an idle shared connection be closed. Shared connection should be enabled.

180000

df_interactive_exec_pool_size

Thread pool size for executing tasks per request in the interactive mode.

10

df_interactive_table_maximum_size

Maximum size of any of the intermediate tables in interactive mode.

10000

df_batch_exec_pool_size

Thread pool size for executing tasks per request in the batch mode.

5

df_sampling_num_threads

Number of sampling tasks to run in parallel.

5

df_interactive_hive_settings

Semi-colon separated list of Entity level Hive configurations which includes the Hive configuration parameters that can be modified.

key1=value1 key2=value2 key3=value3

df_batch_hive_settings

If the builds are slow, use this key to set or change Hive parameters at a pipeline level. These are semi-colon separated list of Entity level Hive configurations.

key1=value1 key2=value2 key3=value3

df_compute_stats_enabled

When enabled, table-statistics are computated for merge or overwrite targets at the end of pipeline build job.

true

PIPELINE_SOURCES_AUTO_SYNC_CHECK_DISABLED

If set to true, it disables the error icon on the pipeline editor page indicating source schema mismatch. If set to false, the error icon displays on the pipeline editor page.


random_sampling_enabled

Determines whether sample generation, for source or pipeline target, needs to be random. To be disabled when sampling is taking too long to complete or is failing.

true

CUBE_DIMENSIONS_ENABLE_DICTIONARY

If dimension tables need to be stored in memory.

true

sqoop_job_map_mem_mb

Mapper mem setting for sqoop. Might need to increase for tables with huge rows.


sqoop_job_red_mem_mb

Reducer mem setting for sqoop.


append_date_to_logfile

Append current date to IW job log file name.

true

LOG_MINER_TABLESPACE_NAME

Table space name.


use_new_tablespace

If we want to start logminer in different tablespace.

false

use_temp_table_for_log_based_cdc

To use temp table approach for log based cdc.

true

TEMP_DATABASE_NAME

Temp database name if we want to use temp database for log based cdc.


build_dictionary_before_cdc

If we want to build database dictionary before every cdc for log-based cdc.

false

oracle_logminer_dictionary_file_name

Oracle dictionary file name.

dictionary.ora

oracle_logminer_dictionary_file_path

Path on oracle server to dictionary file.


SOURCE_TIME_FORMAT

Time format of data for log-based cdc.


SOURCE_DATE_FORMAT

Date format of data for log-based cdc.


sqoop_export_job_map_mem_mb

Mapper mem setting for sqoop. Might need to increase for tables with huge rows.


sqoop_export_job_red_mem_mb

Present export jobs have no reducer, so not necessary now. This configuration will be useful in future.


export_multiplication_factor

Multiplication factor arrays for creation of target table.

2.0,1.5,1.0

shorten_td_export_destination_column_name

Specifies if the column names should be shortened while creating the destination table.

false

shorten_td_export_destination_table_name

Specifies if the table names should be shortened while creating the destination table.

false

td_table_and_column_name_limit

The limit for table name.

128

netezza_export_escape_char

The escape char for stging file in NZ export, only applicable for external tables.

/

netezza_export_enclose_char

The enclose char for stging file in NZ export, only applicable for external tables. Possible options are single quote and double quote.

"

netezza_export_null_value

The null value for stging file in NZ export, only applicable for external tables.

""

netezza_export_should_allow_control_characters_in_data

Should control characters ASCII (0-31) be part of data.

false

fail_job_on_post_hook_failure

If the job should fail when post hook fails.

true

notification_mail_authentication_enabled

For the SMTP server, specifies if authentication is enabled.

true

notification_mail_tls_enabled

For the SMTP server, specifies if TLS is enabled.

true

default_driver_xmx_mb

Default driver memory setting for any Job without the job configurations

512

iw_jobs_default_mr_java_opts_ratio

Default ratio of container xmx to container memory

0.8

iw_jobs_default_mr_map_mem_mb

default map memory for infoworks jobs

2048

iw_jobs_default_mr_red_mem_mb

default reducer memory for infoworks jobs

2048

iw_jobs_default_mr_io_sort_ratio

Default ratio of MR Job io.sort.mb to minimum of mapper and reducer memories

0.3

source_fetch_metadata_rdbms_driver_xmx_mb

Driver Xmx memory for RDBMS Metadata crawl


source_fetch_metadata_sftp_driver_xmx_mb

Driver Xmx memory for SFTP Metadata crawl


source_crawl_rdbms_driver_xmx_mb

Driver Xmx memory for RDBMS and DFI Crawl


source_crawl_sftp_driver_xmx_mb

Driver Xmx memory for SFTP Crawl


source_cdc_driver_xmx_mb

Driver Xmx memory for CDC


source_merge_driver_xmx_mb

Driver Xmx memory for Merge


source_export_driver_xmx_mb

Driver Xmx memory for Export


default_driver_xmx_mb

Driver Xmx memory for Sample Data Generation


pipeline_build_driver_xmx_mb

Driver Xmx memory for DF batch build driver xmx


cube_build_driver_xmx_mb

Driver Xmx memory for Cube build


default_driver_xmx_mb

Driver Xmx memory for Delete entity


source_crawl_cdc_mapmerge_mr_map_mem_mb

Mapper memory CDC


source_crawl_cdc_mapmerge_mr_red_mem_mb

Reducer memory CDC


dfi_job_map_mem

Mapper memory DFI crawl


dfi_job_red_mem

Reducer memory DFI crawl


json_job_map_mem

Mapper memory JSON crawl


json_job_red_mem

Reducer memory JSON crawl


cube_star_schema_job_map_mem_mb

Mapper memory Cube build


cube_star_schema_job_red_mem_mb

Reducer memory Cube build


sqoop_export_job_map_mem_mb

Mapper memory Sqoop Export Data


sqoop_export_job_red_mem_mb

Reducer memory Sqoop Export Data


td_export_mr_map_mem_mb

Mapper memory Teradata Export Data


td_export_mr_red_mem_mb

Reducer memory Teradata Export Data


ADVANCED_ANALYTICS_DISABLED

To disable advanced analytics node, this should be true.

false

XML_ERROR_THRESHHOLD

If the number of error records increases this threshold, the mr job will fail.

100

XML_KEEP_FILES

If the host type is local before the mr job runs, the xml files are copied to the tableId/xml directory. If this config is true, then the files are not deleted after the crawl.

true

xml_job_map_mem

Mapper memory for the crawl map reduce.

Value of iw_jobs_default_mr_map_mem_mb in conf.properties

xml_job_red_mem

Reducer memory for the crawl map reduce.

Value of iw_jobs_default_mr_red_mem_mb in conf.properties

CSV_ERROR_THRESHHOLD

If the number of error records increases this threshold, the mr job will fail.

100

CSV_KEEP_FILES

If the host type is local before the mr job runs, the csv files are copied to the tableId/csv directory. If this config is true, the files are not deleted after the crawl.

true

CSV_TYPE_DETECTION_ROW_COUNT

Number of rows to be read for type detection/metacrawl.

100

CSV_PARSER_LIB

The parser to be used for csv crawl.

Default COMMONS, recommended UNIVOCITY.

CSV_SPLIT_SIZE_MB

Split size to be used for mr for every file.

128

dfi_job_map_mem

Mapper memory for the crawl map reduce.

Value of iw_jobs_default_mr_map_mem_mb in conf.properties

dfi_job_red_mem

Reducer memory for the crawl map reduce.

Value of iw_jobs_default_mr_red_mem_mb in conf.properties

calc_file_level_ing_metrics

If this is set to true, the file level ingestion metrics are calculated at the end of the job.

true

modified_time_as_cksum

If this is true, the modified time is used to determine if the file has been changed or not. If it is set to false, the actual checksum is calculated.

false

delete_table_query_enabled

By default, Delete Query feature is available at the table level. Set IW Constant delete_table_query_enabled to false from UI to hide Delete Query feature.

true

Ufi_max_failure_percentage_per_table

The percentage of files for which the file ingestion is failed before the job is shown as failed.

0.0

fetch_null_timestamped_records

Records having NULL values will be fetched for timestamp based incremental tables during full ingestion if the value is true.

false

MAP_ORACLE_DATE_TO_TIMESTAMP

In earlier versions of Oracle, date can hold timestamps also. If the user wants to depict this into target as well, this constant must be set to true

false

DFI_DELETE_ORIGINAL_FILE_AFTER_DECOMPRESSION

After the file ingestion, the original file will be deleted if this constant is set to true.

False

MIN_ROWS_FOR_MERGEMR

Minimum number of rows in the current data in a secondary partition for the merge. If a secondary partition has number of rows less than this value, the merge jobs will be combined until they reach this threshold.

1000000

pipeline_interactivity_mode

This is a pipeline validation configuration, which when set to 'manual', removes automatic data type validation while saving node properties and enable a Validate option in the node properties page. If this value is set to 'auto', it automatically validates the node data.


cdh_impala_support

Set to true if Impala support is required


fail_ingestion_on_impala_cache_refresh_failure

Set to true if the ingestion job is to be failed in case Impala cache metadata is not refreshed.


iw_hive_ssl_enabled

To Enable or Disable SSL on Hive

false

iw_hive_ssl_truststore_path

Path to Hive Trust Store File Location


iw_hive_ssl_truststore_passwd

Encrypted Password for Trust Store


bcp_crawl_separator

The field separator for bcp import/export


bcp_row_delimiter

The row separator for bcp export


filesystem_scheme

The hdfs path will be pre-populated with the value set in this configuration. The values can be: s3, s3a, s3n, adls, wasb, gs.


df_batch_sparkapp_settings

This configuration is set to overwrite spark configurations like executor memory, driver memory, etc during pipeline batch build.


df_disable_sample_job

This configurations is set to disable sample jobs after pipeline build.

False

df_disable_cache_job

This configurations is set to disable cache jobs after pipeline build.

False

CREATE_DROP_TARGET_SCHEMA

Used to allow Infoworks to run create database and drop database commands. If this value is set to false, the create database and drop database commands will not be executed.

True

target_schema_permission

Used to enable or disable creation of database in pipelines. If this value is set to false, creation of database in pipelines will be disabled.

True

df_dynamic_hive_udf_enabled

Used to enable or disable UDFs on Hive for pipelines. If this value is set to false, UDFs on Hive for pipelines will be disabled.

True

number_of_parallel_jobs_per_entity

The number of jobs that can run in parallel for each entity.

Entity refers to either of the following: Source, Datamodel, Cube.

For example, if this value is set to 3, then a maximum of 3 source crawls or 3 data model builds or 3 cube builds can occur in parallel.

NA

max_number_of_connections

The maximum number of connections allowed between each table and the source, for example, RDBMS.

NA

df_auto_exclusion_enabled

Used to set pipeline target column projection optimization on all nodes.

True

df_merge_exec_pool_size

Number of concurrent tasks to run while performing merge on pipeline targets.

5

df_fail_on_null_or_empty_partition_value

Used to fail the pipeline jobs when partition values are Null or Empty.

False

iw_udfs_dir

Indicates the UDFs directory.

NA

iw_hdfs_prefix

Default HDFS access protocol prefix.

hdfs://

df_hive_analyze_works

Used to enable analyze tables for Hive targets.

True

df_label_auto_cast_enabled

Used to enable Auto Cast mode for Advanced Analytics Model Label Column.

True

df_label_cast_type

Used to indicate the default value for Advanced Analytics Model Label Column.

Double

df_spark_master

Indicates the Spark master mode.

local

df_hive_logging_freq_ms

Used to poll for Hive logs and update on job logs.

60000

df_target_hdfs_cleanup

To retain data on HDFS when the pipeline is removed, set this false.

True

df_batch_spark_settings

Semi-colon separated list of Entity level Hive configurations. Applicable for batch mode pipeline build and to generate sample data in source.

key1=value1;key2=value2;key3=value3

df_scd2_granularity

SCD2 record change granularity on timestamp can be Second,Minute,Hour,Day,Month,Year. This can be set using advance configuration to overwrite configuration for all targets at once.

second

df_custom_udfs_force_copy

Custom UDFs in pipelines are only copied to HDFS when changes are detected. This configuration is used to overwrite changes to custom UDFs.

False

df_disable_sample_job

Used to disable sample job for pipeline targets after pipeline build.

False

df_disable_cache_job

Used to disable cache job for pipeline targets after pipeline build .

False

df_spark_configfile

Spark configuration file path for Interactive mode Spark pipelines.

/etc/spark2/conf/spark-defaults.conf

df_spark_configfile_batch

Spark configuration file path for Batch mode Spark pipelines.

/etc/spark2/conf/spark-defaults.conf

df_batch_spark_coalesce_partitions

Spark coalesce configuration to create lesser files while writing to disk.

NA

df_disable_current_loader

Custom Transformations cannot be loaded from current class loader. To load Custom Transformation from current classloader, set this configuration to False.

True

df_overwrite_log_level

Log Level overwrite at pipeline level. For example, rootLogger=ERROR;io.infoworks=TRACE;infoworks=DEBUG;org.apache.spark=ERROR.

NA

df_dynamic_hive_udf_enabled

Used to disable Hive UDF from loading for every job or request when add jar permissions are disabled. Set this configuration to false when permissions are disabled.

True

df_validation_progress_percent

Used to set the validation process percent in pipeline batch jobs.

10

df_schemasync_progress_percent

Used to set the Schema Sync progress percent in pipeline batch jobs.

10

df_spark_merge_file_num

Spark configuration to merge files using Coalesce option on dataframe during merge process.

1

iw_df_ext_prefix

Pipeline extension prefix.

hdfs://

storage_format

Used to set the default storage format value for pipeline target.

ORC

user_extensions_base_path

Pipeline extension base path.

NA

source_crawl_schema_merge_mr_map_mem_mb

Mapper memory for schema merge jobs.

NA

source_crawl_schema_merge_mr_red_mem_mb

Reducer memory for schema merge jobs.

NA

source_crawl_schema_mapmerge_mr_map_mem_mb

Mapper memory for map-only schema merge jobs.

NA

SUPPORT_RESERVED_KEYWORDS

Used to enable or disable support for reserved keywords. To enables Hive connection on HDP 3.1, this value must be set to false.

True

Admin Configurations Moved from Global to Entity Levels

The following table lists the admin configurations that are moved from global to entity levels:

IWConstant

Current Level

Moved to

UFI_INGEST_HIDDEN_FILES

Global

Table

UFI_MAX_FAILURE_PERCENTAGE_PER_TABLE

Global

Table

EXPORT_PARALLELIZATION_FACTOR

Global

Global

XML_INPUT_FILE_ENCODING

Global

Table

XML_ERROR_THRESHHOLD

Global

Table

XML_KEEP_FILES

Global

Table

FIXED_WIDTH_INPUT_FILE_ENCODING

Global

Table

FIXED_WIDTH_RECORD_SEPARATOR

Global

Table

FIXED_WIDTH_PADDING_CHARACTER

Global

Table

FIXED_WIDTH_PARSER_LIB

Global

Table

MAX_FIXED_WIDTH_RECORD_SIZE

Global

Table

FIXED_WIDTH_THRESHHOLD

Global

Table

FIXED_WIDTH_KEEP_FILES

Global

Table

FIXED_WIDTH_COMMENT_START_CHARACTER

Global

Table

FIXED_WIDTH_SKIP_CHARACTERS_UNTIL_NEW_LINE

Global

Table

FIXED_WIDTH_ROW_ENDS_WITH_NEW_LINE

Global

Table

FIXED_WIDTH_TYPE_DETECTION_ROW_COUNT

Global

Table

sqoopLimit


Source

fetchSize


Source

CDC_START_TIMESTAMP


Table

CDC_END_TIMESTAMP


Table

LOG_BASED_CDC_SPLIT_BY_COLUMN


Table

ORACLE_LOGMINER_DICTIONARY_FILE_PATH


Source

ORACLE_ARCHIVE_LOG_PATH


Source

ORACLE_LOGMINER_DICTIONARY_FILE_NAME


Source

BUILD_DICTIONARY_BEFOR_CDC


Source

TABLE_SCHEMA_FULL_REFRESH


Global

ORACLE_DATE_FUNCTIONS

Global

Global

INCLUDE_CURRENT_LOG

Global

Source

DATABASE_OBJECT_TYPES

Global

Source

TEMP_LOG_TABLE_NAME

Source

Source

LOGMINER_TABLESPACE_NAME


Source

LOGMINER_TEMP_TABLE_TABLESPACE_NAME


Source

CREATE_TEMP_TABLE_PARALLELISM


Source

TEMP_DATABASE_NAME

Source

Source

TEMP_TABLE_INDEX_NAME


Source

CREATE_INDEX_ON_TEMP_TABLE


Source

USE_TEMP_TABLE_FOR_LOG_BASED_CDC

Source

Source

USE_NEW_TABLESPACE_FOR_LOGMINER


Source

IS_WIDE_TABLE

Global

Table

USE_REDO_LOG_DICTIONARY

Source

Source

MAP_ORACLE_DATE_TO_TIMESTAMP

Source

Source

FETCH_NULL_TIMESTAMPED_RECORDS


Source

ORACLE_ARCHIVE_LOG_INFO_OBJECT_NAME

Source

Source

ENABLE_DDL_TRACKING


Source

IS_STAGING_ORACLE_SERVER

Source

Source

DROP_HIVE_SCHEMA

Global

Source

disableCategorical

Global

Source

disableAdditionalTableInfo

Global

Source

disableRowCount

Global

Source

SKIP_RELOADED_CHUNKS

Table

Table

enable_schema_synchronization

Table

Table

validateChunks

Admin

Table

failChunkWhenCountDoesntMatch

Table

Table

CSV_PARSER_LIB

Admin

Table

CSV_SPLIT_SIZE_MB

Admin

Table

CSV_INPUT_FILE_ENCODING

Admin

Table

JSON_INPUT_FILE_ENCODING

Admin

Table

CSV_ERROR_THRESHHOLD

Admin

Table

JSON_ERROR_THRESHHOLD

Admin

Table

CSV_KEEP_FILES

Admin

Table

JSON_KEEP_FILES

Admin

table

CSV_TYPE_DETECTION_ROW_COUNT

Admin

Table

JSON_TYPE_DETECTION_ROW_COUNT

Admin

Table

CONTROL_FILE_READER

Admin

Table

VALIDATE_AFTER_SFTP_TO_LOCAL

Table

Table

VALIDATE_AFTER_LOCAL_TO_HDFS

Admin

Table

SEC_PARTITION_MERGE_RED

Admin

Table

VALIDATE_AFTER_HDFS_TO_HIVE

Admin

Table

CSV_MULTILINE_MODE

Admin

Table

TPT_DEFAULT_CHARSET


Global

TPT_DELIMITER


Table

TPT_FILE_ESCAPE_CHAR


Table

TPT_FILE_QUOTE_ESCAPE_CHAR


Table

TPT_FILE_ENCLOSE_CHAR


Table

TPT_SCRIPT_PATH


Source

TPT_EXPORT_SPOOL_MODE


Table

TPT_CHECKPOINTS_PATH


Source

TPT_LOG_PATH


Source

TPT_CHARACTERSET


Source

TPT_JOB_RESTART_LIMIT


Table

TPT_CHECKPOINT_INTERVAL_IN_SECONDS


Table

TPT_EXPORT_BLOCK_SIZE


Table

TPT_EXPORT_TENACITY_HOURS


Table

TPT_EXPORT_RETRY_INTERVAL_MINS


Table

TPT_IO_BUFFER_SIZE


Table

TPT_HADOOP_BLOCK_SIZE


Table

TPT_EXPORT_MAX_SESSIONS


Table

TPT_EXPORT_MIN_SESSIONS


Table

TPT_EXPORT_OPERATOR_MAX_DECIMALDIGITS


Table

TPT_EXPORT_READER_INSTANCES


Table

TPT_EXPORT_WRITER_INSTANCES


Table

TPT_UNICODE_MULTIPLICATION_FACTOR


Table

TPT_ASCII_MULTIPLICATION_FACTOR


Table

TPT_UTF8_MULTIPLICATION_FACTOR


Table

TPT_UTF16_MULTIPLICATION_FACTOR


Table

USE_TPT_SELECTOR_OPERATOR


Table

IS_TIME_FORAMT_NEEDED


Global

USE_TPT_GENERATED_SCHEMA


Table

ENABLE_TPT_TRACE_LEVEL


Table

USE_TPT_DESTINATION_AS_HDFS


Table

USE_TPT_TDCH_INTERFACE


Table

TPT_FILE_FORMAT


Table

SOURCE_DATE_FORMAT


Source

SOURCE_TIME_FORMAT


Source

SFTP_STAGING_BASE_PATH


Table

SFTP_BUFFER_SIZE


Source

GZIP_FILE_EXTENSION


Source

ENCODE_PRIMARY_PARTITION


Source

DFI_DELETE_ORIGINAL_FILE_AFTER_DECOMPRESSION


Source

export_multiplication_factor


Table/ Pipeline

netezza_export_escape_char


Table/ Pipeline

netezza_export_enclose_char


Table/ Pipeline

netezza_export_null_value


Table/ Pipeline

netezza_export_delimiter


Table/ Pipeline

bcp_crawl_separator


Table/ Pipeline

bcp_row_delimiter


Table/ Pipeline

DF_SNOWFLAKE_VALIDATE_ROW_COUNT


Pipeline

Update password for MongoDB and HIVE

To encrypt and update password for MongoDB or HIVE in the Infoworks system, perform the following:

  • Execute the following interactive bash script: $IW_HOME/apricot-meteor/infoworks_python/infoworks/bin/infoworks_security.sh -encrypt -p

  • Enter the new plain text password when prompted.

  • Copy the encrypted password displayed in the result and update in the Infoworks conf.properties file located in the $IW_HOME/conf folder.

Backup

Overview

This page allows administrators to take a backup of the current metadata stored in MongoDB.

In the System Configuration page, click Backup.


The Metadata Database Backup page has two sections: Backup Schedule, Recent Backups

Backup Schedule

The backup can be taken immediately or it can be scheduled to run whenever required.

The Backup Now button takes an immediate backup of the metadata and is disabled by default until the target path to store the backup is specified.

To specify the target path click Edit Schedule, add absolute target path of local fs, configure any schedule if needed by selecting Enabled in Status dropdown.


Click Save to save the configuration.

Click Backup Now to run the backup metadata job in the background. After the successful completion of the backup, an entry will be added in the Recent Backup view.

Recent Backups

Recent backup view displays the list of all previous backup of the Infoworks metadata.

  • Date: Timestamp on which the backup was taken.

  • Filename: Path of the file where the backup was taken.

  • Status: Whether the backup was successful or not.

Notification

The Notification feature allows the admin to configure emails of success or failure of various jobs, and any issues with Infoworks DWA installation.

NOTE: Email notifications require an SMTP server and an email account to send emails from.

This section describes the steps to configure such notifications.

  • Open the platform-config.json file from the $IW_HOME/ platform/conf folder.

  • Set the following configurations in the messaging-service section as required.

"email.settings": { "smtpHost": "smtp.gmail.com", "smtpPort": 587, "smtpUsername": "postman@infoworks.io", "smtpPassword": "qCfcfcfOtJ9moLFp0XjMhsLuchjdbcq1NMSyr6yzlcXczTJA=", "ttlsEnabled": false, "sslEnabled": false, "sslPort": 465 }
  • Edit the smtpUsername and smtpHost parameters with required values. If the email address provided in the smtpUsername parameter is of a Gmail account, do not make any change to the smtpHost value.

  • Provide the AES encrypted email password for the specified email address as the smtpPassword. If there is no password setup for the specified email id, provide the AES encrypted empty string.

  • To encrypt the password, run the following command : $IW_HOME/apricot-meteor/infoworks_python/infoworks/bin/infoworks_security.sh -encrypt -p <yourpassword>

  • Restart services using the following commands:

$IW_HOME/bin/stop.sh platform

$IW_HOME/bin/start.sh platform