Creating a Source
To create a source, perform the following steps:
- Log in to Infoworks.
- In the left navigation pane, click the Data Sources icon. The Data Sources page appears.
- Click Onboard New Data. The Source Connectors page appears with the list of all available connectors.
- Click the required connector. The Setup Source page of the selected connector appears.
- In the Setup Source page, enter the following details:
| Field | Description | Details |
|---|---|---|
| Source Name | A name for the source that will appear in the Infoworks User Interface. | The source name must be unique and must not contain space or special characters except underscore. For example, Customer_Details. |
| Source File Location | The storage systems location where your files are stored. | You can select one of the following options:
|
| Data Environment | The environment on which the source is defined. | Select the environment from the drop-down list. The list includes the environments created by the Admin. For more details, refer to Managing Environments section. |
| Schema Name | The name of the schema in the data lake. | The name can only contain letters, numbers, hyphen, and underscore. |
| Storage | The target storage where the data will be stored. | Select the storage from the drop-down list. The storage options are displayed based on the environment selected. |
| Base Location | The location of the target on the File System. | The path must start with / and must only contain letters, numbers, hyphen, and underscore. For example, /iw/sources/samplesource. |
6. Click Save.
BigQuery Labels
This label enables you to add labels to your tables in BigQuery.
| Field | Description |
|---|---|
| Key | This field depicts the key for the specific BigQuery label. |
| Value | This field depicts the value for the specific BigQuery label. |
| Description | This field describes the BigQuery label |
| Actions | This field indicates the actions users can perform. For example, Add or delete |
| Add Labels | This fields adds a new BigQuery label |
| Infoworks Managed Labels | If this checkbox is selected, these labels overwrite the BigQuery label. If this checkbox is not selected, these labels will be appended to existing BigQuery labels. |
For more information, refer to BigQuery Labels.
Job Hooks
Job Hooks are external python/bash scripts that can be run before or after a source ingestion job in the data plane. These scripts can access various job properties as environment variables in the data plane. For more information about job hooks, Using a Job Hook