Feature: Support EMR-Serverless workflows

[andrewdkim47]

Have you searched for this feature request?

• I searched but did not find similar requests

Problem Statement

currently we have steps and support for EMR-EC2 EMR Persistent UI.

Similarly, we should add support for EMR-Serverless workflows and understand how customers can plug in the service managed Spark History Server.

Possible Solution

We may be able to utilize

https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_GetDashboardForJobRun.html

Alternatives Considered

No response

[Satyamgupta2365]

Hi @andrewdkim47 @kubeflow team,

I’d like to work on this issue and add EMR Serverless support using the GetDashboardForJobRun API.

My plan:

1. Add a new config option (e.g., emr_serverless_application_id or detect via job run ARN)
2. Implement fetching the dashboard URL via boto3 using get_dashboard_for_job_run
3. Update the MCP server response to return the correct Spark History Server URL for Serverless jobs
4. Add documentation + example for EMR Serverless users

I already have access to an EMR Serverless environment for testing.

Does this approach look good? Any specific preferences or edge cases I should keep in mind before I send a PR?

Thanks!