Troubleshooting

Interactive Service Timeout/Error in Dataproc Environment

If the interactive service times out or an error occurs on the Dataproc environment, perform the following:

1. Log in to the admin console and check if the interactive server is running.
2. If the interactive server is not running, start the interactive server.
3. If the interactive server is running, log in to the edge node machine using SSH and stop the DT service using the following command: /opt/infoworks/bin/stop.sh dt
4. Start the Data Transformation service using the following command: /opt/infoworks/bin/start.sh dt
5. The interactive mode in the Infoworks edge node must work now.

Target Overwrite Failure for Delta Storage Format

If data overwrite fails for the delta storage format, set the following advanced configuration to overwrite the schema for the delta storage format:

Key: dt_spark_df_writer_additional_options

Value: overwriteSchema=true

Pipeline Build Failure for Delta Format on Persistent Cluster

Sometimes, when delta format is used for target node, the pipeline build fails on persistent clusters for Azure Databricks environment. You will receive the following error message "org.apache.spark.sql.AnalysisException: Table or view not found". If you encounter this issue, try restarting the persistent cluster, if it still does not work, build the pipeline again with ephemeral cluster.