Fix Claude analysis gating to handle co-failures correctly

[iangmaia]

Description

Follow-up to #25477. Fixes a logic bug in upload-claude-analysis.sh where the script exited early on the first non-essential failure (e.g. Danger), causing Claude analysis to be silently skipped even when essential jobs also failed in the same build.

Changes:

• Fix co-failure logic: Count non-essential failures first, then query the Buildkite REST API for total failed job count. Only skip Claude when non_essential_failures == total_failures.
• Fail-safe: If the API call fails, Claude runs anyway (safe default).
• Fix shebang: #!/bin/bash -eu → #!/usr/bin/env bash + set -eu for portability.
• Use Bash array: NON_ESSENTIAL_STEPS is now a proper array to avoid word splitting/glob issues.
• YAML literal scalar: custom_prompt: > → custom_prompt: | to preserve numbered list and paragraph structure.
• Include timed_out state: The jq filter catches both failed and timed_out job states.

Test Steps

1. Trigger a build where only Danger fails → Claude analysis should be skipped
2. Trigger a build where Danger and a real test both fail → Claude analysis should run
3. Trigger a build where only a real test fails → Claude analysis should run

---

• I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

[wpmobilebot]

📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.

	App Name	WordPress
	Configuration	Release-Alpha
	Build Number	32005
	Version	PR #25491
	Bundle ID	org.wordpress.alpha
	Commit	c530df9
	Installation URL	166b11m92fe58

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

[wpmobilebot]

📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.

	App Name	Jetpack
	Configuration	Release-Alpha
	Build Number	32005
	Version	PR #25491
	Bundle ID	com.jetpack.alpha
	Commit	c530df9
	Installation URL	333jt4p48oo08

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

[iangmaia]

@mokagio updated this PR as well to check for hard_failed on 683f6ca.

[mokagio]

@iangmaia great job on this update across the apps.

Have you considered a next step where the script goes into CI toolkit with a default array of non-essential jobs comprising of Danger and SwiftLint, and the option for consumers to define a file in the repo (.buildkite/claude-analysis-non-essential-steps?) or an overriding env var in shared-pipeline-vars?

[iangmaia]

Have you considered a next step where the script goes into CI toolkit with a default array of non-essential jobs comprising of Danger and SwiftLint, and the option for consumers to define a file in the repo (.buildkite/claude-analysis-non-essential-steps?) or an overriding env var in shared-pipeline-vars?

That's not a bad idea! Though the Claude Build Analysis is probably going to change to adopt Buildkite Model Providers or a different solution given the plugin has been deprecated. As is, it is still interesting and we can likely migrate to other setup taking the learnings we had so far, but I wouldn't invest in tweaking the current setup that much.

[mokagio]

probably going to change to adopt Buildkite Model Providers or a different solution given the plugin has been deprecated

Good point, good point. I also wonder if this is actually useful for folks?

Do you have any feedback on it? But even then, AI analysis annotations might not be useful for devs but could be useful for us to get to a place where they can be useful? So definitely worth to keep investing on this, the question is how to measure usefulness/effectiveness.

[iangmaia]

Good point, good point. I also wonder if this is actually useful for folks?

Do you have any feedback on it? But even then, AI analysis annotations might not be useful for devs but could be useful for us to get to a place where they can be useful? So definitely worth to keep investing on this, the question is how to measure usefulness/effectiveness.

Yep, I'm still seeing it as an experiment. I often check it, sometimes it has useful insights, but often it is just noisy and verbose, runs when it shouldn't, etc so these changes are important to make it more reliable and relevant. The next iterations (hopefully already in a custom and more compact template) should keep improving it.