Viewing Ingestion Logs
The Jobs page displays the progress and result of the recent build jobs. You can also view the Summary of the build and DB Jobs for each build. Clicking the DB job record displays the Run Details of the record. You can download the run details by clicking the download icon near the status.
Click the Jobs tab to view the source log details and the Download Logs icon to download the source log details.
The Status bar indicated the progress of the build.
Databricks Job Cleanup
Infoworks cleans up the Databricks jobs automatically after a defined period of time. The cleanup job runs every 30 min and by default, the failed jobs are cleaned after 24 hours while the successful jobs are cleaned after 4 hours. The time is measured after the job is finished. This includes the following advanced configurations:
- enable_databricks_job_cleanup: The configuration to enable Databricks job clean up. The default value is true.
- successful_databricks_job_retention_minutes: The configuration to set the period until which the successful Databricks jobs will be retained before cleanup. The default value is 240.
- failed_databricks_job_retention_minutes: The configuration to set the period until which the failed Databricks jobs will be retained before cleanup. The default value is 1440.
This feature ensures that the newly submitted jobs do not fail due to the 1000 jobs limit in Databricks.
Snowflake Environment Job Logs
Every query that gets executed on Snowflake using Infoworks will be tagged with Infoworks_JobId_TableId. During debugging user can search for this tag in Snowflake query console and get all the queries executed for that table in that job. We even log status of each query in the logs and the associated Query Id.
Interactive Job Logs
To view interactive ingestion jobs, navigate to the Interactive Jobs tab within the Jobs section. To download dataplane logs, click the download icon at the end of the relevant row.
