Building a Pipeline

Following are the steps to build a pipeline:

1. Navigate to the Pipeline Design page and click the Build icon.
2. Select the Version and click Build Now to build the pipeline version.
3. You can click the Build Tables Metadata button to enable import of multiple, sequential pipelines with export configuration.

4. Click any of the existing builds under the Recent Builds section to view the detailed information of the job. You can view the summary. You can also download the logs of the job.
5. You can click the Run Restatement Job button to specify upper and lower watermark limits on a pipeline build job run on load incremental source tables. A window will pop up asking the user to input the upper and lower watermark limits for the source tables with load incrementally enabled. Consequently, the records with watermark value > lower_watermark_limit and <= upper_watermark_limit from the source table will be appended to the target table.

NOTE For tables configured in all modes, restatement will occur in 'append' mode.

6. The DB Jobs tab allows you to debug Databricks jobs using the run details.

NOTE Expected to get UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION error while running the pipeline build log on unity catalog enabled cluster with shared access as spark execution engine.