Onboarding Data from Teradata

Select the authentication type from the dropdown. For example, Infoworks Managed or External Secret Store.

If you select Infoworks Managed, then provide Authentication Password for Password.

If you select External Secret Store, then select the Secret which contains the password.

NOTE Only the secrets accessible to the user will be available in the drop-down list.

Check the box to enable ingestion via TPT. For more details, see Onboarding Data from Teradata TPT.

This option cannot be disabled, if any of the tables is configured for TPT(or TPT without IW Processing ). In order to disable this option, you will need to change the "Ingest Using" field setting to JDBC.

The custom query based on which the table has been created.

NOTE This field is only visible if the table is ingested using Add Query as Table.

The combination of keys to uniquely identify the row. This field is mandatory in incremental ingestion tables. It helps in identifying and merging incremental data with the already existing data on target.

NOTE At least one of the columns in the natural key must have a non-null value for Infoworks merge to work.

For Timestamp and Date watermark columns, this option enables an additional offset (decrement) to the starting point for ingested data. Records created or modified within the offset time period are included in the next incremental ingestion job.

NOTE Timestamp watermark column has three options: Days, Hours and Minutes, and the Date watermark column has Days option. In both the cases, the options will be decremented from the starting point.

Select columns on which the target table will be sorted.

NOTE This feature is only available for Snowflake environments.

The name of the target catalog.

NOTE This field will be available only to the Unity Catalog enabled data environments.

The staging schema name for temp tables.

NOTE Applicable only for Unity Catalog enabled environments on Azure Databricks

The format in which the tables must be stored. The options include Read Optimized (Parquet), Read Optimized (ORC), Write Optimized (Avro), Universal Format (UniForm) and Delimited text files.

NOTE Selecting the storage format is not supported in Snowflake environments.

NOTE UniForm is available only to the Unity Catalog enabled data environments.

The column used to partition the data in target. Selecting the Create Derived Column option allows you to derive a column and then use that as the partition column. This option is enabled only if the partition column datatype is date or timestamp.

Provide the Derived Column Function and Derived Column Name. Data will be partitioned based on this derived column.

Select the Create Derived Split Column option and provide the Derived Split Column Function to derive from the Split By column. This option is enabled only if the Split By column data type is integer or timestamp. The data will be split based on the derived value. mod: For every float, integer, or long value, the mod function is performed on the number of connections that are available on the database and based on the mod value the corresponding database is updated. The mod function is first applied to the column and after getting the output the split gets applied and the data is evenly distributed. The mod function is applied on each row level and all the connections run parallelly until the end of the job.

hashamp_hashbucket_hashrow: This function divides the data to the maximum number of amps available in the Teradata. You can apply this function on the columns such as natural key, primary key, and so on. The data is first distributed among the amps and Teradata computes much faster while applying this function. Using hashamp, you can save the time of computing the mod.

Limitation: The hashamp value is restricted to only five amps even though there are several connections running on Teradata.