Script for deleting jobs in Databricks workspace for 1000 job limit workspace

Script for deleting jobs in Databricks workspace for 1000 job limit/workspace

Problem Description

Whenever we submit the Ingestion jobs, the Pipeline jobs or the Export Jobs from Infoworks, Infoworks runs these jobs on the Databricks Job clusters. But there is a 1000 job hard limit per workspace at Databricks side and if the number of jobs submitted from Infoworks exceeds this threshold, the jobs start failing with quota limit exceptions.

Solution

In order to overcome this issue, please run the attached script to clear the completed or failed jobs from the Databricks Job History service.

How to run the script?

To cleanup only infoworks jobs: python databricks_delete_jobs.py -s -t

To cleanup all databricks jobs : python databricks_delete_jobs.py -s -t -j all

source /opt/infoworks/bin/env.sh;python databricks_delete_jobs.py -s https://adb-423464161.1.azuredatabricks.net -t abcdefg-token

Affects version

Infoworks Datafoundry 3.x onwards