Add PyPI Publishing to Automated Release by adnanhemani · Pull Request #4220 · apache/polaris

adnanhemani · 2026-04-16T08:09:47Z

Changes the PyPI nightly build script to use Trusted Publishers, as well as adding publishing scripts when generating a new release candidate and publishing new Polaris versions.

I've tested this from my fork: https://github.com/adnanhemani/polaris/actions/runs/24500027607/

I've already set up Trusted Publishers for both PyPI and TestPyPI:

Result:

Checklist

🛡️ Don't disclose security issues!
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

snazy

Thanks for the effort here!

I cannot judge on the Python/Makefile specific changes.
The changes to the nightly.yml + release-3-... workflows look fine.

I do have two concerns about release-4-...:

The OIDC (id-token: write) permission for PyPi trusted publishing is exposed to all other steps that don't need it.

If, for example, everything but the PyPi publishing fails, it cannot be retried, and we have to re-roll a whole new release including all the vote-"ceremony". Not really new, as Docker-Hub publishing can also fail. I think it's better to have separate workflow-jobs for Maven publishing, Docker-Hub publishing and PyPi publising plus an "aggregator" job that depends on those. This gives us the ability to retry only a failed publishing job without failing a whole release.

(Preexisting) The contents: write permission is only needed to create the GH release. That permission can then also be limited to that "aggregator-job".

snazy · 2026-04-16T12:16:30Z

+    permissions:
+      id-token: write


This permission isn't commonly known, please add a comment explaining what it's used for.

snazy · 2026-04-16T12:16:54Z

+    needs: [prerequisite-checks]
+    permissions:
+      contents: read
+      id-token: write


Same here, deserves a comment.

snazy · 2026-04-16T12:17:41Z

    name: Generate Release Email Body
    runs-on: ubuntu-latest
-    needs: [prerequisite-checks, build-and-publish-artifacts, build-docker, build-and-stage-helm-chart]
+    needs: [prerequisite-checks, build-and-publish-artifacts, build-docker, build-and-stage-helm-chart, publish-python-client-rc]


Nit: the list is growing, maybe migrate to one line per dependency.

snazy · 2026-04-16T12:19:08Z

    runs-on: ubuntu-latest
    permissions:
      contents: write
+      id-token: write


This one concerns me, because all steps now get access to OIDC/trusted-publishing.

+1 we can move this to the step that pushes to pypi

HonahX

@adnanhemani Thank you so much for adding this to our release process!

HonahX · 2026-04-16T16:18:19Z

+	uv version "$(BUILD_VERSION)" && \
+	uv build --clear


Instead of making an internal one, could we re-use client-build and extend it to optionally set the version?
Currently, CLI's License & Notice are only valid if we only publish the wheel (it will be a universal wheel): #3891 (comment). So we probably need FORMAT=wheel when building artifacts

Good call, changed.

kevinjqliu · 2026-04-16T17:48:44Z

+    permissions:
+      id-token: write


Suggested change

permissions:

id-token: write

permissions:

# IMPORTANT: this permission is mandatory for Trusted Publishing

id-token: write

https://docs.pypi.org/trusted-publishers/using-a-publisher/

kevinjqliu

looks great, thanks!

for pyiceberg, we push RCs as pre-release to pypi. we should double check that its indeed labeled as pre-release. Otherwise, if it accidentally gets label as the normal release, we'd have to yank the entire version (and publish a patch release). very tedious!

adnanhemani

Thank you all for your reviews. At this time, I've only separated out the "publish-python-client" step in the release-4... script, as it is the smallest change relevant to this PR. We should continue the refactor as @snazy suggests in a different PR.

adnanhemani · 2026-04-16T22:20:48Z

+    permissions:
+      id-token: write


adnanhemani · 2026-04-16T22:20:56Z

+    permissions:
+      id-token: write


adnanhemani · 2026-04-16T22:21:18Z

+    needs: [prerequisite-checks]
+    permissions:
+      contents: read
+      id-token: write


adnanhemani · 2026-04-16T22:21:27Z

    name: Generate Release Email Body
    runs-on: ubuntu-latest
-    needs: [prerequisite-checks, build-and-publish-artifacts, build-docker, build-and-stage-helm-chart]
+    needs: [prerequisite-checks, build-and-publish-artifacts, build-docker, build-and-stage-helm-chart, publish-python-client-rc]


adnanhemani · 2026-04-16T22:22:02Z

    runs-on: ubuntu-latest
    permissions:
      contents: write
+      id-token: write


adnanhemani · 2026-04-16T22:22:24Z

+	uv version "$(BUILD_VERSION)" && \
+	uv build --clear


Good call, changed.

adnanhemani · 2026-04-16T22:52:13Z

@kevinjqliu - we are pushing RCs to TestPyPI and only pushing the released versions to PyPI. I believe this should be good enough to not cause errant versions?

kevinjqliu

Minor bug when passing the format arg, otherwise LGTM

Applied the fix locally and tested:

make client-nightly-build
RC_VERSION=1.4.0 RC_NUMBER=1 make client-rc-build                                                                    
RELEASE_VERSION=1.4.0 make client-release-build

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

snazy · 2026-04-21T11:02:03Z

The recent Polaris 1.4.0 release workflow failed quite early during the changes to Apache SVN, before it handles Helm charts, Docker images or Nexus or even create the GitHub release. I think, Helm charts, Docker images and Nexus jobs would have worked fine during that workflow run, if those would have been separate jobs in the workflow.

The unsolvable issue with release-related workflows is that those steps cannot be integration tested. This is why I think splitting the work across multiple jobs reduces the amount of non-executed steps, that require manual work later.

Adding PyPi publishing adds another step that can potentially fail a whole release, but having that as a separate workflow job reduces the risk of affecting other parts of the release automation.

adnanhemani · 2026-04-21T20:20:58Z

@snazy I agree with you - that's why I separated the PyPI uploads into separate Jobs. Did I miss something there?

I also don't disagree with your comments that we should separate all these steps into different jobs - but I don't find that necessary in this PR. I would highly prefer to do that in a follow up PR to reduce the scope of this PR. Is this a blocker for you?

adnanhemani added 7 commits April 15, 2026 18:50

Add PyPI Publishing to Automated Release

9c73901

Add PyPI Publishing to Automated Release

e58cf72

remove fork guard

68aa6dd

fix spurrious change

d69c708

fix up for PR

4e76f71

refactor

2d2e7bf

fork changes

e0c7bcf

github-project-automation Bot added this to Basic Kanban Board Apr 16, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Apr 16, 2026

github-advanced-security AI found potential problems Apr 16, 2026

View reviewed changes

Comment thread .github/workflows/nightly.yml Fixed

Comment thread .github/workflows/release-3-build-and-publish-artifacts.yml Fixed

Comment thread .github/workflows/release-4-publish-release.yml Fixed

adnanhemani added 2 commits April 16, 2026 01:16

zizmor and bug fix

2a5d327

updating to allowlisted action version

2eba4bc

adnanhemani marked this pull request as ready for review April 16, 2026 08:28

revert testing changes

893be43

snazy reviewed Apr 16, 2026

View reviewed changes

HonahX reviewed Apr 16, 2026

View reviewed changes

kevinjqliu reviewed Apr 16, 2026

View reviewed changes

review comments

0bcd755

adnanhemani commented Apr 16, 2026

View reviewed changes

kevinjqliu reviewed Apr 17, 2026

View reviewed changes

Comment thread .github/workflows/nightly.yml

kevinjqliu reviewed Apr 17, 2026

View reviewed changes

Comment thread Makefile Outdated

kevinjqliu approved these changes Apr 17, 2026

View reviewed changes

adnanhemani and others added 3 commits April 17, 2026 17:18

Update Makefile

ee7b105

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

Upload Python Wheel file to SVN, Use SVN to upload to PyPI

116e0f7

ensure both wheel and source files are uploaded

0d0812d

Conversation

adnanhemani commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX left a comment

Choose a reason for hiding this comment

Uh oh!

HonahX Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

adnanhemani left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adnanhemani commented Apr 16, 2026

Uh oh!

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

snazy commented Apr 21, 2026

Uh oh!

adnanhemani commented Apr 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

adnanhemani commented Apr 16, 2026 •

edited

Loading

HonahX Apr 16, 2026 •

edited

Loading