Configuring Infoworks with Amazon EMR

Introduction

Amazon EMR is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop or Apache Spark, on AWS to process data for analytics purposes and business intelligence workloads.

The EMR role defines the allowable actions for Amazon EMR while provisioning resources. You can perform service-level tasks that are not performed in the context of an EC2 instance running within a cluster. For example, the service role is used to provision EC2 instances when a cluster launches.

NOTE The default role is EMR_DefaultRole. The Amazon EMR scoped default managed policy attached to EMR_DefaultRole is AmazonEMRServicePolicy_v2. The v2 policy replaces the default managed policy, AmazonElasticMapReduceRole, which is on the path to deprecation.

NOTE The AmazonEMRServicePolicy_v2 depends on scoped down access to resources that EMR provisions or uses.

To use this policy, you must:

1. Pass the user tag for-use-with-amazon-emr-managed-policies = true when provisioning the cluster. The EMR automatically propagates the tags.
2. Additionally, you must manually add a user tag to specific types of resources, such as EC2 security groups that are not created by EMR. For more information on tagging resources, refer to Tagging resources to use managed policies.

Prerequisites

• During installation, ensure that EMR default roles are present.
• Create access key and secret key for an IAM user with the AmazonElasticMapreduceFullAccess and ElasticMapreduceFullAccesspolicyV2 privileges and full access to the Staging Bucket, Warehouse Bucket, and Target Bucket. For providing access to the buckets, refer to Granting Permissions—Amazon Simple Storage Service.
• The cluster subnet must be whitelisted for 3000 port on the Edge node.
• Ensure that the EC2 Instance Profile used to create compute has access to AWS glue account.
  • To use glue as a metastore in different AWS accounts, cross account access is required. Ensure that EC2 Instance Profile has appropriate glue permissions. For more details, refer to Granting Cross-Account Access.
  • To use glue as a metastore in the same AWS account, add EC2 Instance Profile to IAM policy of AWS Glue.

Procedure

To configure and connect to the required Amazon EMR instance, navigate to Admin > Manage Data Environments, and then click Add button under the Amazon EMR option.

The following window appears:

There are three tabs to be configured as follows:

• Data Environment
• Compute
• Storage

Data Environment

To configure the data environment details, enter values in the following fields. This defines the environmental parameters, to allow Infoworks to be configured to the required Amazon EMR instance:

After entering all the required values, click Continue to move to the Compute tab.

Compute

A Compute template is the infrastructure used to execute a job. This compute infrastructure requires access to the metastore and storage that needs to be processed.To configure the compute details, enter values in the following fields. This defines the compute template parameters, to allow Infoworks to be configured to the required Amazon EMR instance.

Infoworks supports creating multiple persistent clusters in an Amazon EMR environment, by clicking on Add button.

Enter the fields in the Compute section:

After entering all the required values, click Continue to move to the Storage tab.

NOTE When autoscale is enabled on EMR clusters, verify that the replication factor on HDFS is configured correctly as per the EMR documentation. If replication is not sufficient, you might observe issues with spark jobs when clusters are downscaled.

Storage

To configure the storage details, enter values in the following fields. This defines the storage parameters, to allow Infoworks to be configured to the required Amazon EMR instance:

NOTE To configure a new storage after the first time configuration, click Add button on the UI.

Enter the following fields under the Storage section:

After entering all the required values, click Save. Click Finish to view and access the list of all the environments configured. Edit, Clone, and Delete actions are available on the UI, corresponding to every configured environment.

Add a New Runtime Version for Amazon EMR

To add a new runtime version for Amazon EMR, perform the following steps:

1. SSH into Infoworks machine.
2. Navigate to the path where Infoworks is installed. For example: IW_HOME, /opt/infoworks/
3. Open conf/emr_defaults.json file.
4. Under the Runtime version section, add the following.

xxxxxxxxxx

1  {

2  "label": "6.2.0",

3  "runtime_version": "emr-6.2.0",

4  "ml_runtime_version": "emr-6.2.0",

5  "scala": "2.12",

6  "spark": "3.0.1",

7  "python": "3.8"

8  }

NOTE Ensure to add the right version of the environment, which you are using for runtime_version, spark, and scala.

Else, the job will fail.

5. Save and close the file.

NOTE Validate and ensure that there are no errors in the JSON file before saving.

LIMITATIONS

• To view the limitations while using AWS Glue, refer to Considerations when using AWS Glue Data Catalog.
• To run the jobs on ephemeral clusters due to rate limits on AWS EMR: as per the AWS EMR documentation, there are specific limits for EMR APIs to work around the EMR rate limits. Infoworks uses a custom Job Submission Driver which works on the persistent cluster, but ephemeral cluster will still encounter rate limit related errors. This is a known limitation on EMR ephemeral clusters at higher submission rates.
• To avoid hitting the EMR API limit, the Infoworks job status will now be updated directly from the Spark application. However, if the Spark application is terminated outside of Infoworks, the job status in Infoworks will remain stuck in a "running" state. Therefore, users should cancel EMR jobs through Infoworks only.