Introduction

Feature Highlights

Following are some data preparation feature highlights:

• User-friendly GUI-driven method to transform data.
• SQL syntax support for transforming data.
• ETL workload import support for importing required SQL files to automatically build data pipelines.
• Immediate feedback on syntax and semantic errors.
• Enterprise-friendly features, including domain-based source access and audits.
• Multiple pipelines to create workflows.
• Interactive data preparation for large data sets.
• Intelligent data sampling to view data changes as transformations applied.
• Visual debugging using interactive data.
• Advance analytics nodes.

Optimizations

Data preparation provides the following optimizations:

Design-Time Optimizations

• Instant feedback for data, syntax and semantic errors: During transformation, data errors like data format in column, regex on columns having issues, etc can be verified in the sample data.
• Support to visualize flow of data to design better flows.
• Auto-materialize transformation nodes for faster responses.
• Automatic dependency management: When a transformation node is modified, the system automatically computes the dependent nodes. The platform uses a Mark and Sweep algorithm to perform this efficiently.
• Safe handling, refactors column include/exclude/rename even in user-defined expressions.
• Automatic rename of duplicate column names.
• Reuse of Hive and Impala connections to support interactive viewing of data while designing pipelines.

Execution Optimizations

• Automatic detection of pipeline-specific intermediate storage by backend engine.
• Automatic parallelization of population of multiple targets.
• Expression Reuse: computes once, uses multiple times, reducing CPU/IO.
• Optimal merge process to handle updates.
• Ability to use pipeline-specific environment settings for MapReduce, Hive, memory, and compression for build.
• Automatic selection of only required columns based on the pipeline and thus reduce CPU/IO usage.

Query-Time Optimizations

• Primary partitions with tables created through pipeline targets.
• Different Format ORC and Parquet on pipeline targets.
• Table statistics update after every build for Cost-based Optimizer on pipeline targets.