Performing Custom Transformation

Prerequisites

Perform the following:

• Navigate to the IW_HOME/opt/infoworks/conf/conf.properties file.
• Set the extension path in the user_extensions_base_path configuration. For example, user_extensions_base_path=/opt/infoworks/extensions/.
• Ensure that the Infoworks user has write access to the folder.
• Restart the UI and Data Transformation services.

Creating Custom Transformation

• Implement the SparkCustomTransformation API with a class which has a null constructor. For example, SparkCustomTfmSample.java in IW_HOME/examples/custom-transformation/java/sample-code in the Infoworks edge node. You can modify this example java class with the specific transformation requirements. The interfaces to be implemented are packaged in dt-extensions-1.0.jar which you must import in the project. The dt-extensions-1.0.jar is packaged and available with the Infoworks package in the IW_HOME/lib/dt/ folder.
• Create a jar for the above project.
• Upload the jar (and any external dependencies such as Pheonix jars) to a folder in the Infoworks edge node. For example, /home//pivottransformation/. Jars related to Hadoop, spark etc, will already be available on the Infoworks classpath and hence, not required to be placed in this folder.

NOTES

• The Custom Transformation node is not supported for pipelines in snowflake environment.
• For the custom target logs to be available in the downloaded cluster logs from the Infoworks custom target pipeline build page, the custom target package should be named io.infoworks.CustomTarget

Registering Custom Transformation

To register custom transformation, perform the following steps:

1. Navigate to Admin > Extensions > Pipeline Extensions.
2. Click Create Pipeline Extension.

In the Add Pipeline Extension page, enter the following details:

3. Click Save. Once extension is added successfully, the above transformations will be available in Spark pipeline custom transformations node.

Adding Custom Transformation to Domain

To add custom transformation to domain:

1. Navigate to Admin > Domains.
2. Click the Manage Artifacts button for the required domain.
3. Click the Add Pipeline Extensions button in the Accessible Pipeline Extensions section.
4. Select the Custom Transformation to be made accessible in the domain.

Using Custom Transformation

Following are the steps to apply Custom Transformation node in pipeline:

1. Open the pipeline editor, drag and drop Custom Transformation and link it with the dependent parent nodes.
2. Double-click Custom Transformation and click Edit Properties.
3. In the Properties page, select the appropriate Extension Name and Extension Class from the respective drop-down lists.
4. Enter a Key and Value which will be added to the transformation node and will be available via UserProperties class from the public void initialiseContext(SparkSession sparkSession, UserProperties userProperties, ProcessingContext processingContext) API.
5. Click Save.

NOTE When using custom transformation, the output columns must be manually added.

Python Custom Transformation

Creating Python Custom Transformation

NOTE Python Custom Transformations are not supported for K8s based installations.

To create Python Custom Transformation:

1. Install the python package in IW_HOME/lib/dt/api/python/dt-extensions-python-api-1.0.egg using the following command: python -m easy_install <path_for_api-1.0.egg>
2. Write the custom transformation logic by implementing the Python API CustomTransformation. To import, add the following statement to the Python custom logic: from api.custom_transformation import CustomTransformation
3. Example for custom transformation is available at IW_HOME/ examples/custom-transformation/python/sample-code
4. Create .egg extension file for the python project.

NOTE Use the Java APIs to apply transformation on the input datasets. Internally, py4j is used to communicate to Python custom logic. And, all the provided input data frames are in Java Object form.

Registering Python Custom Transformation

To register Python Custom Transformation:

1. In Infoworks, navigate to Admin > Extensions > Pipeline Extensions.
2. Click Create Pipeline Extension.
3. In the Add Pipeline Extension page, enter the following details:

NOTES

• Py4j and Pyspark packages must be installed on Python.
• Custom transformation with Python runtime is only supported from Python v2.7 onwards.

Configurations

• Add the dt_socket_port_range configuration in the pipeline advanced configuration to provide available port range (which will be internally used for socket communication). For example, 25810 : 25890. The default range is 25400 to 25500.
• Add the dt_socket_max_tries in pipeline advanced configuration to set the maximum number of tries to establish the socket communication. For example, 8. The default value is 10.
• Set the Python path by adding following configuration to the IW_HOME/conf/conf.properties file python_custom_executable_path=<python_executable_path>

NOTE If the above configuration is set, the path will be used for Python custom logic. If not set, the default Python version installed on that node will be used.