Skip to content

E2E: Fix false-negative WooCommerce login assertion, remove dead Apple specs#110012

Merged
lucatume merged 7 commits intotrunkfrom
fix/QAO-104-apple-authentication-woocommerce-user
Apr 20, 2026
Merged

E2E: Fix false-negative WooCommerce login assertion, remove dead Apple specs#110012
lucatume merged 7 commits intotrunkfrom
fix/QAO-104-apple-authentication-woocommerce-user

Conversation

@lucatume
Copy link
Copy Markdown
Contributor

@lucatume lucatume commented Apr 16, 2026

Part of QAO-104

Proposed Changes

  • Strengthen the WooCommerce TOTP test's final assertion (authentication__totp.spec.ts:111-115) with a Log Out link visibility check that proves the session is real, not just a URL match
  • Delete authentication__apple.spec.ts: both tests inside it were unconditionally test.skip(true, ...) since Sept 2025 (PR e2e tests: skip Apple authentication tests #106099), and the WooCommerce case duplicates coverage already provided by the TOTP spec
  • Delete the legacy Jest-runner duplicate signup__woo-apple.ts (quarantined, scenario covered by Playwright Test specs)
  • Align the post-add-to-cart wait in DomainSearchComponent.selectSuggestion so the Continue button wait uses a 30s timeout, matching the preceding Add to cart detach wait

Why are these changes being made?

QAO-104 reported a false-negative in the Apple-for-WooCommerce E2E test: the final check only verified a redirect URL to woocommerce.com, which passes even when the user is not actually authenticated. Investigation found that:

  1. The same URL-only weakness exists in the TOTP-for-WooCommerce test, which is the one actually running in CI (the Apple test has been fully skipped since Sept 2025 due to Apple account lockouts).
  2. No ready-made Apple sign-in mock exists, and the WPCOM backend validates Apple JWTs against Apple's real JWKS with no non-production override path.
  3. The Apple WooCommerce case duplicates the WPCC handshake already exercised by the TOTP test.

This PR fixes the live test (TOTP) with a DOM-level logged-in check and removes the dead Apple specs. Backend JWT validation coverage is added in a companion wpcom PR: 211934-ghe-Automattic/wpcom

E2E stabilization: DomainSearchComponent.selectSuggestion Continue-button wait

While reviewing CI for this PR, the two flows/setup-domain__flows.spec.ts specs ("create a free site and then add a domain WITHOUT a plan upgrade" and "create a paid site, add a domain, then cancel the plan") failed as new failures on desktop Chrome and mobile Pixel with:

TimeoutError: locator.waitFor: Timeout 10000ms exceeded.
  - waiting for getByRole('listitem').first().getByRole('button', { name: 'Continue' }) to be visible
  at DomainSearchComponent.selectSuggestion (domain-search-component.ts:200)

The error-context DOM snapshot captured at the moment of failure showed the Continue button was present in the first list item — it simply rendered after the default 10s timeout. The preceding step in the same method waits up to 30s for the Add to cart button to detach, so the short 10s window for the replacement Continue button to appear was the tight spot on slow CI.

Fix: pass { timeout: 30000 } to the Continue waitFor, matching the upstream detach wait. No behavior change on fast paths.

Testing Instructions

  • Run the TOTP authentication spec in headed mode against production and verify the Log out locator resolves on woocommerce.com/my-dashboard/
  • Run yarn playwright test specs/flows/setup-domain__flows.spec.ts --reporter=list --repeat-each=5 against a local Calypso and confirm all runs pass on both desktop Chrome and mobile Pixel
  • Confirm no type errors: npx tsc --noEmit in test/e2e/
  • Confirm no lint errors: yarn eslint test/e2e/specs/authentication/authentication__totp.spec.ts packages/calypso-e2e/src/lib/components/domain-search-component.ts

…pple specs

The WooCommerce TOTP test asserted only a URL match on /my-dashboard/,
which passes even when the user is not actually authenticated (QAO-104).
Add a Log Out link visibility check that proves the session is real.

Delete authentication__apple.spec.ts: both tests inside it have been
unconditionally skipped since Sept 2025 (account gets locked on Apple)
and the WooCommerce case duplicates coverage already provided by the
TOTP spec. Backend JWT validation coverage moves to a wpcom PHPUnit
isolated test in a separate PR.

Delete the legacy Jest-runner duplicate signup__woo-apple.ts (quarantined,
scenario already covered by the Playwright Test specs).
@lucatume lucatume self-assigned this Apr 16, 2026
lucatume and others added 4 commits April 16, 2026 16:03
DomainSearchComponent.selectSuggestion waits up to 30s for the
Add-to-cart button to detach, but only 10s (the default) for the
Continue button to appear on the same row. On slow CI runs the
Continue button renders after 10s but well under 30s, causing
setup-domain flow tests to fail with the Continue button actually
present in the DOM snapshot captured at the moment of failure.

Align the two waits at 30s.
…-woocommerce-user' into fix/QAO-104-apple-authentication-woocommerce-user
@lucatume lucatume marked this pull request as ready for review April 17, 2026 10:20
@lucatume lucatume requested a review from alopezari April 17, 2026 10:21
@matticbot matticbot added the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Apr 17, 2026
@lucatume
Copy link
Copy Markdown
Contributor Author

I've added, in this PR, a fix for a test that would consistently fail in CI.
While not about the code touched here, it's unblocking the PR and providing benefit.

@matticbot
Copy link
Copy Markdown
Contributor

This PR modifies the release build for the following Calypso Apps:

For info about this notification, see here: PCYsg-OT6-p2

  • agents-manager
  • help-center
  • notifications
  • wpcom-block-editor

To test WordPress.com changes, run install-plugin.sh $pluginSlug fix/QAO-104-apple-authentication-woocommerce-user on your sandbox.

Copy link
Copy Markdown
Contributor

@alopezari alopezari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @lucatume!

LGTM, just let's keep monitoring these tests in case extending the timeout doesn't solve the issues.

As a note, I've run the tests according to the testing instructions and there's one that failed:

[pixel] › specs/flows/setup-domain__flows.spec.ts:613:7 › Setup Domain Flows › As a new user, I can transfer an external domain to a new site @calypso-release › And I select the Personal plan

I ran it against staging, and it didn't fail because of the test, but because the site got stuck for more than 1 minute trying to register a domain:

video.webm

Could this be related to Calypso Live? Maybe it's a silly question, but still I'm not 100% sure about the scope of Calypso Live, it's an area where I haven't worked much yet.

Let me approve it because the PR adds good improvements anyway.

@lucatume lucatume merged commit 65f5021 into trunk Apr 20, 2026
14 checks passed
@lucatume lucatume deleted the fix/QAO-104-apple-authentication-woocommerce-user branch April 20, 2026 09:14
@github-actions github-actions bot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants