Monitor active job drivers for EMR cluster

Monitor active job drivers for EMR cluster

Below is a script to monitor for active EMR job drivers that start with "ClusterJobDriver_". At any given time there should be only one "ClusterJobDriver_" in the running state.

The below script is to calculate the no of active "ClusterJobDriver_ " and notify by email or AWS SNS topic when the count is more than 1.

  • Need to configure AWS region and cluster ID in the script

  • The script assumes aws cli is already setup with the required authentication

  • All the job drivers launched for ingestion or pipeline jobs start with "ClusterJobDriver_".

#!/bin/bash

# Set the AWS region and EMR cluster ID
AWS_REGION="your_aws_region"
EMR_CLUSTER_ID="your_emr_cluster_id"

# Get the list of step functions for the EMR cluster that start with "ClusterJobDriver_"

step_functions=$(aws emr list-steps --cluster-id "$EMR_CLUSTER_ID" --region "$AWS_REGION" --output json | jq '.Steps[] | select(.Name | startswith("ClusterJobDriver_")) | select(.Status.State == "RUNNING")')

# Count the number of running step functions
running_count=$(echo "$step_functions" | jq 'length')

# Display the count of running step functions
echo "Running Step Functions Count: $running_count"

# Check if the running count is more than 1
if [[ $running_count -gt 1 ]]; then
# Email alert subject
alert_subject="More than 1 job driver found to be in running state on Cluster $EMR_CLUSTER_ID"


Next to read:
Optimizing Ingestion Performance and Reducing Cluster Costs Using Split-By and Table Group Parallelism in Infoworks

For more details, refer to our Knowledge Base and Best Practices!

For help, contact our support team!

© UNIPHORE TECHNOLOGIES 2025 | Confidential

On This Page
Monitor active job drivers for EMR clusterMonitor active job drivers for EMR cluster