Creating a Pipeline Group
Overview
Pipelines can be grouped together in a Pipeline Group to run as a single job in Infoworks. Pipelines built in a pipeline group use a single connection and run as one transaction.
From the pipeline group, you can configure the following:
- Data environment
- Pipeline group name, description
- Environment details
- Pipelines(including pipeline version) to be included in the group
- Execution order of pipelines added to the group
Even if one pipeline job fails, it would result in rollback of already executed pipelines in the respective group.
Pipeline Group Creation
To create a new pipeline group:
Step 1: Go to Domains and select the specific domain.
Step 2: Click New Pipeline Group.
Step 3: Provide the following pipeline group details.
| Field | Description | Details |
|---|---|---|
| Data Environment | Name of the data environment | Select the name from the dropdown |
| Name | Name of the pipeline group | |
| Description | Description of the pipeline group | - |
| Snowflake Warehouse | Snowflake warehouse name. | Snowflake Warehouse drop-down will appear based on the selected snowflake profile. |
| Snowflake Profile | Snowflake profile name. | Snowflake Profile drop-down will appear for the selected snowflake environment. |
| Run driver job on data plane | Select this checkbox to run the job driver on data plane. | |
| Compute Cluster | The compute cluster that is spin up for each table. | |
| Custom Tags | They're key-value pairs that help you identify resources based on settings that are applied to your cloud resources | |
| Associated Custom Tags | The custom tags selected appear here. | |
| Query Tags | A string that is added to the Snowflake query tag and can be accessed via Query history in Snowflake. |
Under the All Pipelines tab, add the required pipelines to the pipeline group.
| Field | Description | Details |
|---|---|---|
| Name | Name of the pipeline | |
| Created By | User who created the pipeline | |
| Created At | Timestamp for pipeline Creation | |
| Added to the Group | Whether it is added to any pipeline group |
Step 4: Click Add to add the pipelines to the pipeline group.
Step 5: Under the Pipelines in the Group tab, configure the following fields based on your requirement:
| Field | Description |
|---|---|
| Name | Name of the pipeline. |
| Run Active Version | Select this checkbox to run the active version |
| Version Number | Active version is the default version number. If you want to run any other version, select the specific version from the dropdown. |
| Execution Order | Describes the order in which the pipelines will get executed. |
Step 6: Click Save.
- Creating volatile temporary tables is not supported for pipelines running in a group.
- Running multiple pipeline groups in parallel on databricks interactive cluster is not recommended since they will share the static session context and might result in unwanted results.
- Target table data will be truncated in case of overwrite pipelines when the pipeline group fails.
Pipeline Groups Audits
Pipeline Groups auditing is available at two levels:
- Global Pipeline Groups Audit
At the global level, the audit logs capture key pipeline groups - level events such as:
- Pipeline groups creation
- Pipeline groups updates
- Pipeline groups deletion
These events are accessible via the “View Audits” button at the top right in the Pipeline Groups tab.
- Individual Pipeline Groups Audit
Each pipeline groups also has its own detailed audit trail. This includes:
Field-level tracking for all creates, updates, and deletes operations
Capturing what specific values were added, modified, or removed from individual fields
To access these:
- Navigate to the “Action” column in the specific pipeline group.
- Click the circular arrow icon (next to the delete icon).
- This opens a detailed view showing what field values were added, changed, or removed within that specific pipeline group.
