Skip to content

Fix #27107: soft-deleted users still appear in Experts/Reviewers across all entities#27120

Open
yan-3005 wants to merge 30 commits intomainfrom
ram/fix-27107-soft-deleted-users-in-relations
Open

Fix #27107: soft-deleted users still appear in Experts/Reviewers across all entities#27120
yan-3005 wants to merge 30 commits intomainfrom
ram/fix-27107-soft-deleted-users-in-relations

Conversation

@yan-3005
Copy link
Copy Markdown
Contributor

@yan-3005 yan-3005 commented Apr 7, 2026

Summary

Fixes #27107 — soft-deleted users appear in `owners`, `experts`, `reviewers`, `followers`, and `votes` fields on bulk-list API responses (`GET /entities?fields=...`) across every entity type.


Root Cause

The bug traces back to an inverted ternary in two methods in `Entity.java` (`getEntityReferenceById` and `getEntityReferencesByIds`):

```java
// WRONG (was on main for both methods)
include = repository.supportsSoftDelete ? Include.ALL : include;
// Reads: "if the type supports soft delete, ignore caller's include, force ALL"
// That's backwards.

// CORRECT (this fix)
include = repository.supportsSoftDelete ? include : Include.ALL;
// Reads: "if the type has no deleted column, force ALL (filtering impossible); else respect caller"
```

History of workarounds

PR #25284 (Jan 2026) discovered this bug and worked around it by adding two new methods:
`getEntityReferenceByIdRespectingInclude` / `getEntityReferencesByIdsRespectingInclude`
with the correct ternary — but the original broken methods were never fixed. Two method pairs, same signature, one broken, one correct — a permanent trap.

Earlier in this branch, a ThreadLocal `bulkInclude` approach was tried: thread the `include` parameter from API callers through virtual dispatch into `setFieldsInBulk` → `fetchAndSetFields` → `fetchAndSetRelationshipFieldsInBulk` → `resolveRelationshipEntityReferencesByType`. This was another workaround layer on top of the existing one, not a root fix. Rejected and removed.


The Fix

1. Fix the root — flip both ternaries in `Entity.java`

The two entity-reference resolution methods now correctly respect the caller's `include` for soft-delete-supporting types, and fall back to `ALL` only for types without a `deleted` column (DOMAIN, DATA_PRODUCT, CLASSIFICATION, TAG, etc.) where filtering is not possible.

2. Delete the `RespectingInclude` pair

After the ternary flip, `getEntityReferenceByIdRespectingInclude` and `getEntityReferencesByIdsRespectingInclude` are identical to the originals. They're deleted. Call sites in `EntityRelationshipRepository` are migrated back to the original names.

3. Bulk-list resolver hardcodes `NON_DELETED` — semantic, not a hack

In `resolveRelationshipEntityReferencesByType` and all five `batchFetch*` methods (owners, followers, reviewers, experts, votes), nested reference hydration is unconditionally `NON_DELETED`.

Why this is correct, not a hack:

  • `?include=` is a top-level filter — it controls which entities appear in the list (e.g., `GET /glossaries?include=all` returns soft-deleted glossaries)
  • It does not change the semantics of nested relationship pointers, which are always pointers to live entities
  • A soft-deleted user is not a "live" expert/owner/reviewer of anything
  • For the "show me deleted relationships on a specific entity" use case, single-entity GET with `?includeRelations=field:all` (PR Filter deleted users from ownership relationships in GET operations #25284 feature) is the correct endpoint — unaffected by this change

The ThreadLocal approach (deleted) was trying to pass the top-level `include` down to nested resolution, which is the wrong semantic regardless of implementation.

4. Fix null `include` on single-entity GET for some entity types

`DomainResource`, `DataProductResource`, `EventSubscriptionResource`, `QueryResource` do not declare `@QueryParam("include")`. When these endpoints are called, `include` is `null`, which inside `RelationIncludes` defaulted to `Include.ALL` — leaking soft-deleted nested refs on single-entity GET.

Fixed by defaulting `null → Include.NON_DELETED` before constructing `RelationIncludes` in `EntityResource.getInternal` and `getByNameInternal`.


Behavioral Changes vs Main

Scenario Before After
Bulk `GET /entities?fields=owners` + soft-deleted owner leaks ❌ filtered ✅
Bulk `GET /entities?fields=experts` + soft-deleted expert leaks ❌ filtered ✅
Bulk `GET /entities?fields=reviewers` + soft-deleted reviewer leaks ❌ filtered ✅
Bulk `GET /entities?fields=followers` + soft-deleted follower leaks ❌ filtered ✅
Bulk `GET /entities?fields=votes` + soft-deleted voter leaks ❌ filtered ✅
Bulk `?include=all` + soft-deleted nested person ref leaks filtered (semantic: include=all applies to top-level list only)
Single-entity GET `GET /domains/{id}?fields=experts` + soft-deleted expert leaks ❌ (null include bug) filtered ✅
Single-entity GET (Table, GlossaryTerm, etc.) already correct ✅ unchanged ✅
`?fields=domains` / `?fields=dataProducts` (no deleted column) ALL always ALL always (unchanged)

Files Changed

File What changed
`Entity.java` Flip both ternaries; delete `getEntityReferenceByIdRespectingInclude` + `getEntityReferencesByIdsRespectingInclude`
`EntityRelationshipRepository.java` 2 call sites: rename from `RespectingInclude` variants to standard methods (now identical after ternary fix)
`EntityRepository.java` Remove ThreadLocal `bulkInclude`; 3-arg `setFieldsInBulk` overload delegates to 2-arg (no threading); `resolveRelationshipEntityReferencesByType` and all five `batchFetch*` use `NON_DELETED`
`EntityResource.java` Default `null` include to `NON_DELETED` in `getInternal` / `getByNameInternal`
`DataProductRepository.java` `batchFetchExperts`: rename `getEntityReferencesByIdsRespectingInclude` → `getEntityReferencesByIds`
`DomainRepository.java` Same rename as DataProductRepository
`EntityRepositoryIncludeThreadingTest.java` Deleted — tested the ThreadLocal approach we rejected
`DataProductResourceIT.java` `softDeletedExpert_notReturnedInSingleGet`, `_listEndpoint`, `softDeletedOwner_notReturnedInListEndpoint`
`DomainResourceIT.java` `softDeletedExpert_notReturnedInSingleGet`, `_listEndpoint`, `softDeletedExpert_notReturnedInListWithIncludeAll`
`GlossaryTermResourceIT.java` `softDeletedReviewer_notReturnedInListEndpoint`

Test Plan

  • `mvn spotless:apply` — clean
  • `DomainResourceIT` — 3/3 pass (singleGet, listEndpoint, listWithIncludeAll)
  • `DataProductResourceIT` — 3/3 pass (singleGet, listEndpoint, softDeletedOwner)
  • `GlossaryTermResourceIT` — 1/1 pass (softDeletedReviewer_listEndpoint)

Known Out of Scope

Search-index propagation: soft-deleting a user does not update embedded `EntityReference` copies in other entities' Elasticsearch/OpenSearch documents. Explore-page / search-box results may still surface the soft-deleted user until a reindex runs. Separate PR.


Summary by Gitar

  • EntityRepository cleanup:
    • Removed unused setFieldsInBulk(fields, entities, include) overload to simplify bulk field resolution.
    • Added a null guard in batchFetchFollowers to prevent NPEs when resolving follower references.
  • Testing additions:
    • Added integration tests for Domain entities to verify soft-deleted followers and voters are correctly filtered in list endpoints.

This will update automatically on new commits.

…ss all entities

Two bugs caused soft-deleted users to leak into experts/reviewers/owners/followers:

1. EntityRepository.resolveRelationshipEntityReferencesByType hardcoded Include.ALL
   for the consolidated bulk list path, affecting ALL entities (DataProduct, Domain,
   GlossaryTerm, Glossary, Table, Schema, Service, Container, etc.). Fixed by
   switching to getEntityReferencesByIdsRespectingInclude(..., NON_DELETED).

2. EntityResource.getInternal / getByNameInternal built RelationIncludes(null, ...)
   which defaulted to Include.ALL, causing single-entity GET on DataProduct, Domain,
   EventSubscription, Query to return deleted users in relation fields. Fixed by
   defaulting null include to Include.NON_DELETED for all resource endpoints.

Also cleaned up dead-code in DataProductRepository.batchFetchExperts and
DomainRepository.batchFetchExperts which used getEntityReferenceById that
silently overrides NON_DELETED to ALL.

Adds regression tests in DataProductResourceIT, DomainResourceIT,
GlossaryTermResourceIT covering both single-GET and list-endpoint paths.
@yan-3005 yan-3005 added safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch backend labels Apr 7, 2026
Copilot AI review requested due to automatic review settings April 7, 2026 09:21
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes soft-deleted users appearing in relationship-backed fields (Experts/Reviewers/Owners/Followers/etc.) by ensuring relationship reference resolution defaults to Include.NON_DELETED in both list and single-entity GET paths, and adds integration coverage to prevent regressions.

Changes:

  • Default null include to Include.NON_DELETED when building RelationIncludes in EntityResource GET flows.
  • Update bulk relationship reference resolution to use Entity.getEntityReferencesByIdsRespectingInclude(..., Include.NON_DELETED) so soft-deleted related entities are filtered out.
  • Refactor Domain/DataProduct expert batch loaders to batch-resolve user references while respecting soft-delete, plus add integration tests covering list + single GET behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
openmetadata-service/src/main/java/org/openmetadata/service/resources/EntityResource.java Ensures RelationIncludes defaults to non-deleted when include is absent, fixing single-GET exposure in affected resources.
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java Filters soft-deleted related entity references during bulk field hydration for list endpoints.
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/DomainRepository.java Batch-fetch experts via include-respecting reference resolution and skip deleted users.
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/DataProductRepository.java Same expert batch-fetch fix as DomainRepository for DataProducts.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/GlossaryTermResourceIT.java Adds regression test ensuring soft-deleted reviewers don’t appear in list responses.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DomainResourceIT.java Adds regression tests for soft-deleted experts excluded from single GET and list.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DataProductResourceIT.java Adds regression tests for soft-deleted experts excluded from single GET and list.

…ding NON_DELETED

- Add Include parameter to resolveRelationshipEntityReferencesByType, fetchAndSetRelationshipFieldsInBulk, and fetchAndSetFields (new 3-arg overload)
- Add setFieldsInBulk(fields, entities, Include) overload in base class; existing 2-arg delegates to NON_DELETED
- The Include is no longer hardcoded — callers with request context (e.g. include=all) can pass it through via the 3-arg setFieldsInBulk or fetchAndSetFields

Fixes: #27107
… Domain

DataProductRepository.setInheritedFields and DomainRepository.setInheritedFields
both fetched parent Domain entities with Include.ALL, causing the domain's
experts and owners fields to be resolved with ALL include — returning
soft-deleted users. Changed to NON_DELETED so inherited fields only carry
active users.

Fixes: #27107
Copilot AI review requested due to automatic review settings April 7, 2026 11:32
…ll entities

Five additional bug sites using Include.ALL when resolving user relationships:

- EntityRepository.batchFetchOwners: getEntityReferencesByIds → getEntityReferencesByIdsRespectingInclude(NON_DELETED)
- EntityRepository.batchFetchReviewers: same fix
- EntityRepository.batchFetchExperts (base class): same fix
- TagRepository.setInheritedFields: fetch Classification with NON_DELETED (was ALL), preventing soft-deleted owners/reviewers from being inherited by Tags
- WorksheetRepository.setInheritedFields: fetch Spreadsheet with NON_DELETED (was ALL), preventing soft-deleted owners from being inherited by Worksheets

Fixes: #27107
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comment on lines +1881 to +1885
public void setFieldsInBulk(Fields fields, List<T> entities) {
setFieldsInBulk(fields, entities, Include.NON_DELETED);
}

public void setFieldsInBulk(Fields fields, List<T> entities, Include include) {
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new setFieldsInBulk(..., Include include) overload isn’t used by the core list/batch paths (e.g., listAfter/listBefore and get(ids, ..., include) still call setFieldsInBulk(fields, entities), which hard-defaults to NON_DELETED). This makes list endpoints that accept include=all/deleted inconsistent with single-entity GET, where include affects relation loading via RelationIncludes. Consider plumbing the ListFilter/include value through to setFieldsInBulk so relationship references respect the requested include (while still defaulting null->NON_DELETED).

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

yan-3005 added 2 commits April 8, 2026 21:08
setFieldsInBulk, fetchAndSetFields, fetchAndSetRelationshipFieldsInBulk, and
resolveRelationshipEntityReferencesByType now accept an Include parameter so
that include=all list queries correctly surface soft-deleted relationship
targets (owners, experts, reviewers) — matching the single-entity GET path
behaviour via RelationIncludes.

All existing 2-param overrides delegate to the new 3-param overloads with
NON_DELETED, so all subclass repositories and the update hydration path are
unchanged. All list/get callers with access to an Include value (listAfter,
listBefore, listAll, listAfterKeyset, listWithOffset, get, getByNames) now
forward that value through the chain.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comment on lines 1885 to 1891
public void setFieldsInBulk(Fields fields, List<T> entities, Include include) {
if (entities == null || entities.isEmpty()) {
return;
}
try (var ignored = phase("fetchFields")) {
fetchAndSetFields(entities, fields);
fetchAndSetFields(entities, fields, include);
}
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setFieldsInBulk(fields, entities, include) passes include straight through to fetchAndSetFields(..., include) / resolveRelationshipEntityReferencesByType(..., include). However include is allowed to be null in several call sites (e.g., new ListFilter(null) is used in multiple resources/background workflows), which will cause relationship reference resolution to effectively behave like Include.ALL (soft-deleted related users reappear) and may also propagate a null Include into repository DAO methods.

Consider normalizing include at the boundary (e.g., Include resolved = include != null ? include : Include.NON_DELETED;) and using resolved for the bulk relationship reference resolution path, so the default behavior remains “non-deleted” even when callers omit include.

Copilot uses AI. Check for mistakes.
yan-3005 and others added 2 commits April 8, 2026 22:08
… setFieldsInBulk dispatch

The previous approach made callers invoke the 3-param setFieldsInBulk directly,
which bypassed all subclass overrides of the 2-param version (GlossaryTermRepository,
DatabaseRepository, TableRepository, etc. all override 2-param with essential
hydration logic like populateParentAndGlossaryReferencesInBulk). This caused a
regression where entities from those repos would be missing service references,
parent references, and other custom bulk-hydrated fields.

Fix: store the include in a static ThreadLocal before delegating to the virtual
2-param setFieldsInBulk (so all subclass overrides run), then the 2-param
fetchAndSetFields reads bulkInclude.get() so it reaches
fetchAndSetRelationshipFieldsInBulk with the correct value. The 3-param overload
is now final to prevent further accidental overrides.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comment on lines +1900 to +1907
public final void setFieldsInBulk(Fields fields, List<T> entities, Include include) {
Include previous = bulkInclude.get();
bulkInclude.set(include);
try {
setFieldsInBulk(fields, entities);
} finally {
bulkInclude.set(previous);
}
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include can be null (e.g., several callers build new ListFilter(null)), but this method stores the null in bulkInclude and the relationship bulk loader forwards it into Entity.getEntityReferencesByIdsRespectingInclude(...). For soft-deletable types like user, a null include means the deleted filter is not applied, which can re-introduce soft-deleted users in owners/experts/reviewers/followers (and may lead to inconsistent behavior vs API defaults).

Resolve include == null to a concrete value (likely Include.NON_DELETED, matching the resource-layer defaulting you added) before setting the ThreadLocal.

Copilot uses AI. Check for mistakes.
yan-3005 and others added 2 commits April 9, 2026 12:00
Callers like DomainResource/DataProductResource/PersonaResource build
new ListFilter(null), so filter.getInclude() returns null. That null was
stored in the bulkInclude ThreadLocal and propagated into
getEntityReferencesByIdsRespectingInclude, bypassing the soft-delete
filter for USER/TEAM types and re-introducing soft-deleted users in
owners/experts/reviewers/followers on those list endpoints.

Resolve null to NON_DELETED before setting the ThreadLocal.
Add unit test covering null → NON_DELETED defaulting.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

@sonika-shah
Copy link
Copy Markdown
Collaborator

Review: Changes in EntityRepository and EntityResource can silently break existing behavior

The fix for filtering soft-deleted users from relationships is correct in intent. However, the changes in EntityRepository and EntityResource introduce a NON_DELETED fallback (include != null ? include : Include.NON_DELETED) and a ThreadLocal with NON_DELETED as default. These changes are not required for the fix and will silently change behavior for internal code paths that were working correctly.


1. EntityResource changes are unnecessary

Both getByNameInternal and getInternal add:

Include resolvedInclude = include != null ? include : Include.NON_DELETED;

But include is never null from API calls — every endpoint declares:

@DefaultValue("non-deleted") Include include

The 5 resources that explicitly pass null (DataProductResource, DomainResource, QueryResource, EventSubscriptionResource, DataInsightSystemChartResource) do so intentionally. This is a core function used by 70+ resources — adding a silent default here is risky for zero benefit.


2. ThreadLocal in EntityRepository.setFieldsInBulk silently changes behavior for internal callers

The ThreadLocal<Include> bulkInclude defaults to NON_DELETED. The 2-arg setFieldsInBulk(fields, entities) is called by many internal code paths (not just APIs). These callers never had an include filter applied to relationship resolution before. Now they silently get NON_DELETED.

Scenario A: Bulk PUT hydration breaks

EntityRepository.processBulkPutBatch() (line ~9907) calls:

setFieldsInBulk(putFields, originalsForHydration);  // 2-arg → ThreadLocal = NON_DELETED

This hydrates existing entities to diff against incoming updates. If an entity has a soft-deleted owner, that owner is now silently filtered out. The diff sees "owner was added" instead of "owner was changed" → wrong change events, potential data loss during bulk CSV import.

Scenario B: Cross-repo calls get wrong Include

Several repository overrides fetch related entities with ALL and then call another repository's 2-arg setFieldsInBulk:

  • TestCaseRepository:218-223 — fetches tables with ALL, then calls tableRepo.setFieldsInBulk(fields, tables) → table owners/domains resolved with NON_DELETED instead of ALL
  • TagRepository:277-283 — fetches classifications with ALL, then calls classificationRepo.setFieldsInBulk(fields, classifications) → classification owners/reviewers resolved with NON_DELETED
  • WorksheetRepository:254-256 — fetches spreadsheets with ALL, then calls spreadsheetRepo.setFieldsInBulk(fields, spreadsheets) → spreadsheet owners/domains resolved with NON_DELETED
  • StoredProcedureRepository:163-165 — fetches schemas with ALL, then calls schemaRepository.setFieldsInBulk(fields, schemas) → schema relationships resolved with NON_DELETED

In all these cases, the entity is fetched with ALL but its relationships are now silently filtered with NON_DELETED. Inconsistent state.

Scenario C: GlossaryTerm search ignores API include parameter

GlossaryTermRepository.searchGlossaryTermsInternal (line ~2555) receives include from the API but never passes it to setFieldsInBulk:

setFieldsInBulk(getFields(fieldsParam), terms);  // 2-arg → ThreadLocal = NON_DELETED

If a user queries with include=all, glossary terms themselves are returned, but their owners/reviewers/experts are filtered to NON_DELETED. API contract is broken — user asks for all, gets partial relationship data.

Scenario D: Reindex works by accident

During reindex, PaginatedEntitiesSource uses ListFilter(Include.ALL), which flows through the 3-arg setFieldsInBulk and sets the ThreadLocal to ALL. Cross-repo calls inside subclass overrides then read ALL from the ThreadLocal — correct, but only by coincidence. If the execution order changes or a new code path is added, it silently breaks.


Suggestion

The actual fix (replacing Entity.getEntityReferencesByIds(..., ALL) with Entity.getEntityReferencesByIdsRespectingInclude(..., NON_DELETED) for owners, reviewers, experts) is the right approach and lives in EntityRepository's internal methods. That part is good.

The EntityResource null-guard and the ThreadLocal default are not needed for this fix and introduce risk. Consider:

  • Removing the EntityResource changes entirely
  • Having the ThreadLocal default to ALL (preserving existing behavior) instead of NON_DELETED, so that internal callers that don't set it explicitly continue to work as before
  • Or passing include explicitly through the internal method chain (fetchAndSetFieldsfetchAndSetRelationshipFieldsInBulk) without the ThreadLocal — these are private/protected methods inside EntityRepository and don't require changing the 30 subclass overrides

@yan-3005
Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review @sonika-shah. Addressing each point:

1. EntityResource null guard

The 5 resources that pass null (DomainResource, DataProductResource, QueryResource, EventSubscriptionResource, DataInsightSystemChartResource) don't declare @QueryParam("include") — they pass null explicitly to getInternal/getByNameInternal.

Without the guard, null flows into new RelationIncludes(null), which defaults to Include.ALL (EntityUtil.java:603: this.defaultInclude = defaultInclude != null ? defaultInclude : ALL). This means soft-deleted users would still appear as owners/reviewers/experts in single-GET responses for those resources — exactly the bug we're fixing.

The guard is a no-op for the 70+ resources that declare @DefaultValue("non-deleted"), since include is never null for them.

2. ThreadLocal default — keeping NON_DELETED

The pre-PR hardcoded ALL in resolveRelationshipEntityReferencesByType was the root cause of #27107. Defaulting the ThreadLocal to ALL would leave the bug unfixed for internal 2-arg callers whose data is exposed to users via API:

Scenario B re-examined — cross-repo calls ARE part of the bug path:

  • TestCaseRepository:220 — calls tableRepo.setFieldsInBulk(fields, tables) (2-arg). The hydrated tables (including their owners/domains) are returned in test case API responses. With ALL default, soft-deleted owners would still appear.
  • TagRepository:282 — calls classificationRepository.setFieldsInBulk(fields, classifications) (2-arg). Classification owners/reviewers are exposed to users through tag responses.
  • WorksheetRepository:256 — same pattern with spreadsheet owners/domains.
  • StoredProcedureRepository:165 — uses EntityUtil.Fields.EMPTY_FIELDS (empty set), so no relationship fields are loaded. ThreadLocal is irrelevant here.

These aren't "internal-only" paths — they resolve relationship data that reaches API consumers.

Scenario A (bulkUpdateEntities): Minor diff semantics change — "owner added" vs "owner changed" when the original's owner is soft-deleted. End state is always correct, no data loss. The soft-deleted owner is semantically gone.

Scenario C (GlossaryTerm search): Pre-existing gap — searchGlossaryTermsInternal has include in scope but doesn't pass it to setFieldsInBulk. Not caused by our PR. The NON_DELETED default actually fixes this gap.

Scenario D (reindex): Uses 3-arg setFieldsInBulk(fields, entities, filter.getInclude()) with ALL explicitly. Unaffected by the default.

All API list paths (listAfter, listBefore, listAll, get(ids), getByNames, serializeJsons) use the 3-arg overload with the caller's explicit Include, so they respect include=all when requested.

@github-actions
Copy link
Copy Markdown
Contributor

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

…leted-users-in-relations

# Conflicts:
#	openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

(entityType, ownerIds) -> {
var ownerRefs =
Entity.getEntityReferencesByIds(entityType, new ArrayList<>(ownerIds), ALL);
Entity.getEntityReferencesByIdsRespectingInclude(
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batchFetchOwners, batchFetchReviewers, and batchFetchExperts now always filter relationship targets with NON_DELETED, which can contradict the stated behavior that include=all list queries should surface soft-deleted relationship targets (to match single-entity GET behavior). If these batch loaders can run on list/bulk hydration paths, they should use the request’s Include (e.g., pass include into these helpers, or read the same resolved include used for bulk hydration) instead of hardcoding NON_DELETED.

Suggested change
Entity.getEntityReferencesByIdsRespectingInclude(
entityType, new ArrayList<>(ownerIds), ALL);

Copilot uses AI. Check for mistakes.
Comment on lines +9288 to +9289
var reviewerRefs =
Entity.getEntityReferencesByIds(entityType, new ArrayList<>(reviewerIds), ALL);
Entity.getEntityReferencesByIdsRespectingInclude(
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batchFetchOwners, batchFetchReviewers, and batchFetchExperts now always filter relationship targets with NON_DELETED, which can contradict the stated behavior that include=all list queries should surface soft-deleted relationship targets (to match single-entity GET behavior). If these batch loaders can run on list/bulk hydration paths, they should use the request’s Include (e.g., pass include into these helpers, or read the same resolved include used for bulk hydration) instead of hardcoding NON_DELETED.

Copilot uses AI. Check for mistakes.

// Batch fetch all expert references
// Batch fetch all expert references, filtering out soft-deleted users
Map<UUID, EntityReference> expertRefs =
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batchFetchOwners, batchFetchReviewers, and batchFetchExperts now always filter relationship targets with NON_DELETED, which can contradict the stated behavior that include=all list queries should surface soft-deleted relationship targets (to match single-entity GET behavior). If these batch loaders can run on list/bulk hydration paths, they should use the request’s Include (e.g., pass include into these helpers, or read the same resolved include used for bulk hydration) instead of hardcoding NON_DELETED.

Copilot uses AI. Check for mistakes.
Comment on lines +363 to +364
private static final ThreadLocal<Include> bulkInclude =
ThreadLocal.withInitial(() -> Include.NON_DELETED);
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a static ThreadLocal to implicitly carry Include makes the include behavior non-obvious and more fragile (e.g., harder to reason about, and easy to bypass if any code path calls fetchAndSetFields(..., fields) without going through the setFieldsInBulk(..., include) wrapper). A more maintainable approach is to pass Include explicitly through the internal call chain (or centralize the implementation so both overloads call a single non-overridable method that takes Include) and avoid hidden thread-local state.

Suggested change
private static final ThreadLocal<Include> bulkInclude =
ThreadLocal.withInitial(() -> Include.NON_DELETED);
private static final Include DEFAULT_BULK_INCLUDE = Include.NON_DELETED;

Copilot uses AI. Check for mistakes.
Comment on lines +1914 to +1923
public final void setFieldsInBulk(Fields fields, List<T> entities, Include include) {
Include resolved = include != null ? include : Include.NON_DELETED;
Include previous = bulkInclude.get();
bulkInclude.set(resolved);
try {
setFieldsInBulk(fields, entities);
} finally {
bulkInclude.set(previous);
}
}
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a static ThreadLocal to implicitly carry Include makes the include behavior non-obvious and more fragile (e.g., harder to reason about, and easy to bypass if any code path calls fetchAndSetFields(..., fields) without going through the setFieldsInBulk(..., include) wrapper). A more maintainable approach is to pass Include explicitly through the internal call chain (or centralize the implementation so both overloads call a single non-overridable method that takes Include) and avoid hidden thread-local state.

Copilot uses AI. Check for mistakes.
}
}

protected void fetchAndSetFields(List<T> entities, Fields fields) {
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a static ThreadLocal to implicitly carry Include makes the include behavior non-obvious and more fragile (e.g., harder to reason about, and easy to bypass if any code path calls fetchAndSetFields(..., fields) without going through the setFieldsInBulk(..., include) wrapper). A more maintainable approach is to pass Include explicitly through the internal call chain (or centralize the implementation so both overloads call a single non-overridable method that takes Include) and avoid hidden thread-local state.

Suggested change
protected void fetchAndSetFields(List<T> entities, Fields fields) {
fetchAndSetFields(entities, fields, NON_DELETED);

Copilot uses AI. Check for mistakes.
Comment on lines +1218 to +1230
Domain listed = null;
ListParams params = new ListParams().setFields("experts").withLimit(100);
while (listed == null) {
ListResponse<Domain> page = listEntities(params);
listed =
page.getData().stream()
.filter(d -> d.getId().equals(domain.getId()))
.findFirst()
.orElse(null);
String after = page.getPaging() != null ? page.getPaging().getAfter() : null;
if (listed != null || after == null) break;
params = new ListParams().setFields("experts").withLimit(100).setAfter(after);
}
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paging loop has no hard stop / cursor-repeat protection. If paging ever returns a non-null after that doesn’t advance (or cycles), this test can hang indefinitely in CI. Consider adding a max-iterations bound and/or tracking previously seen after cursors to fail fast with a clear assertion.

Copilot uses AI. Check for mistakes.
yan-3005 and others added 2 commits April 20, 2026 14:31
…ETED

Root cause: Entity.java had an inverted ternary in getEntityReferenceById and
getEntityReferencesByIds:

  // WRONG — forces ALL when type supports soft delete
  include = repository.supportsSoftDelete ? Include.ALL : include;

  // CORRECT — respects caller's include; falls back to ALL only when the
  //            type has no deleted column and filtering isn't possible
  include = repository.supportsSoftDelete ? include : Include.ALL;

PR #25284 worked around this by introducing getEntityReferenceByIdRespectingInclude /
getEntityReferencesByIdsRespectingInclude with the correct ternary, but the original
broken methods were never fixed — a permanent trap. This commit fixes the root and
deletes the RespectingInclude pair (now identical after the flip).

For the bulk-list path (GET /entities?fields=...) nested reference hydration is
hardcoded to NON_DELETED unconditionally in resolveRelationshipEntityReferencesByType
and all five batchFetch* methods (owners, followers, reviewers, experts, votes).
The ?include= query param controls which top-level entities appear in the list;
it does not change the semantics of nested relationship pointers, which should
always resolve to live entities.

For single-entity GET: EntityResource.getInternal/getByNameInternal had a latent
null-include bug for resources (Domain, DataProduct, EventSubscription, Query) that
don't declare @QueryParam("include"). Those null values defaulted to ALL inside
RelationIncludes, leaking soft-deleted nested refs. Fixed by defaulting null →
NON_DELETED before constructing RelationIncludes.

Remove ThreadLocal bulkInclude and the include-threading approach added earlier in
this branch — superseded by the root-cause fix and the NON_DELETED semantic above.
Delete EntityRepositoryIncludeThreadingTest (tested the rejected approach).

Tests: 7 integration tests covering single-GET and list endpoints for Domain,
DataProduct, and GlossaryTerm; verify soft-deleted experts/owners/reviewers are
absent from both paths, including with ?include=all on the list endpoint.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Comment on lines 9083 to 9087
Map<UUID, EntityReference> followerRefs =
Entity.getEntityReferencesByIds(USER, followerIds, ALL).stream()
Entity.getEntityReferencesByIds(USER, followerIds, NON_DELETED).stream()
.collect(Collectors.toMap(EntityReference::getId, Function.identity()));

records.forEach(
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batchFetchFollowers now fetches follower refs with Include.NON_DELETED, which means followerRefs.get(followerId) can legitimately return null (e.g., follower user was soft-deleted). The current loop adds that null reference into the followers list, which can lead to null entries in API responses and downstream NPEs/serialization issues. Skip/omit the follower when the reference is missing (and consider de-duplicating followerIds to avoid unnecessary fetch work).

Copilot uses AI. Check for mistakes.
}

public final void setFieldsInBulk(Fields fields, List<T> entities, Include include) {
setFieldsInBulk(fields, entities);
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new setFieldsInBulk(fields, entities, include) overload ignores its include parameter and simply delegates to the 2-arg method. Since callers were updated to pass include, this reads as if bulk field hydration depends on include when it currently does not. Consider removing this overload and reverting call sites, or (if include is meant to affect bulk hydration) plumb it through where needed instead of accepting an unused parameter.

Suggested change
setFieldsInBulk(fields, entities);
if (entities == null || entities.isEmpty()) {
return;
}
for (T entity : entities) {
setFields(entity, fields, include);
}

Copilot uses AI. Check for mistakes.
Comment on lines 9127 to 9131
Map<UUID, EntityReference> userRefs =
Entity.getEntityReferencesByIds(Entity.USER, new ArrayList<>(allUserIds), ALL).stream()
Entity.getEntityReferencesByIds(Entity.USER, new ArrayList<>(allUserIds), NON_DELETED)
.stream()
.collect(Collectors.toMap(EntityReference::getId, Function.identity()));

Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes vote hydration filter voters with Include.NON_DELETED. There are new integration tests for owners/experts/reviewers, but I don't see coverage specifically asserting that soft-deleted users are excluded from votes in list/bulk responses. Adding an IT for the votes field (and similarly for followers) would help prevent regressions for the original issue scope.

Copilot generated this review using guidance from repository custom instructions.
…arg setFieldsInBulk, add follower/voter IT tests

- batchFetchFollowers: add null-check before adding followerRef to list.
  When a follower is soft-deleted, getEntityReferencesByIds(NON_DELETED)
  omits their ID from the result map, so followerRefs.get(followerId)
  returns null. All other batchFetch* methods already guard this;
  batchFetchFollowers was the only one missing the check.

- Remove the dead 3-arg setFieldsInBulk(Fields, List<T>, Include) overload
  that accepted include but silently discarded it, delegating to the 2-arg
  form. Revert all 6 call sites back to the 2-arg form. The overload was
  misleading — callers appeared to plumb include through but bulk hydration
  was never affected; NON_DELETED is hardcoded in resolveRelationship-
  EntityReferencesByType.

- Add integration tests for followers and votes:
  softDeletedFollower_notReturnedInListEndpoint and
  softDeletedVoter_notReturnedInListEndpoint in DomainResourceIT.
  Both pass (5/5 DomainResourceIT soft-delete tests green).
@sonarqubecloud
Copy link
Copy Markdown

@gitar-bot
Copy link
Copy Markdown

gitar-bot bot commented Apr 20, 2026

Reviewing your code

Code Review ✅ Approved 5 resolved / 5 findings

Implements proper filtration of soft-deleted users in bulk operations and expert listings. This resolves issues with hardcoded deletion flags, duplicate map collisions, and improper include parameter propagation.

✅ 5 resolved
Bug: Hardcoded NON_DELETED ignores caller's include=all in bulk path

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:8707-8708
In resolveRelationshipEntityReferencesByType, Include.NON_DELETED is hardcoded. This method is called from the bulk/list path (fetchAndSetRelationshipFieldsInBulk) which does not propagate the caller's Include parameter. When an admin requests ?include=all on a list endpoint, the entities themselves will include soft-deleted ones, but their relationship fields (experts, reviewers, owners, etc.) will silently drop references to soft-deleted users.

The single-entity GET path (setFieldsInternal) does respect the caller's Include via RelationIncludes, so the behavior is inconsistent between single GET and list endpoints.

For the purpose of this PR (fixing the default case where soft-deleted users leak into normal views), this is correct. But it introduces a secondary issue for admin/audit views that intentionally request deleted entities. Consider threading the Include parameter through fetchAndSetRelationshipFieldsInBulkresolveRelationshipEntityReferencesByType so that include=all still works for list operations.

Bug: Collectors.toMap without merge function may throw on duplicates

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/DataProductRepository.java:987 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/DomainRepository.java:597
In DataProductRepository.batchFetchExperts (line 987) and DomainRepository.batchFetchExperts (line 597), Collectors.toMap(EntityReference::getId, Function.identity()) is used without a merge function. While the input IDs are .distinct(), if getEntityReferencesByIdsRespectingInclude were to return duplicate references for the same ID (e.g., due to a data inconsistency), this would throw IllegalStateException. The generic resolveRelationshipEntityReferencesByType at line 8720 already uses (a, b) -> a as a merge function.

Bug: setFieldsInBulk Include parameter not threaded from callers

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1882 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1885 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:8428 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:8431
The new setFieldsInBulk(Fields, List<T>, Include) overload was added to thread the Include parameter through bulk relationship resolution. However, no caller actually passes a non-default Include value. All existing call sites (list operations in get(), getByNames(), etc.) use the 2-parameter overload which hardcodes Include.NON_DELETED.

This means that when a caller explicitly requests include=all (e.g., an admin wanting to see soft-deleted related entities), the bulk list path will still filter them out. The single-entity get() path at EntityRepository lines 1322-1329 receives an Include parameter but does not forward it to setFieldsInBulk. If the intent of this PR is to always filter soft-deleted from relationships, this is fine and the 3-parameter overload is dead code that should be removed for clarity. If the intent is to respect the caller's include parameter, the threading is incomplete.

Bug: 3-param setFieldsInBulk bypasses all subclass overrides

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1881-1895 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1325 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1768 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1850 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1934 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:2025 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:9388
All callers in this diff changed from setFieldsInBulk(fields, entities) (2-param) to setFieldsInBulk(fields, entities, include) (3-param). However, ~20 subclasses (TableRepository, DashboardRepository, GlossaryTermRepository, DomainRepository, DatabaseRepository, TopicRepository, PipelineRepository, ChartRepository, MlModelRepository, ContainerRepository, etc.) override the 2-param version with essential custom logic (e.g. fetchAndSetDefaultService, fetchAndSetDefaultFields, populateParentAndGlossaryReferencesInBulk). Since none of them override the new 3-param version, Java dispatches directly to the base class implementation, completely skipping all subclass custom hydration logic.

This means list/get operations for Tables, Dashboards, GlossaryTerms, and many other entities will return entities missing service references, default fields, parent references, etc. — a severe regression across the entire API.

Suggested fix: Instead of adding a new overload, thread the Include parameter through the existing overridable method. Either:

  1. Change the existing 2-param signature to 3-param in the base class and all subclass overrides, or
  2. Store include in a field/ThreadLocal before calling the 2-param method so subclass overrides still execute, or
  3. Keep the new 3-param method but have it call this.setFieldsInBulk(fields, entities) (the virtual 2-param dispatch) for the subclass logic, then separately handle the include filtering in a non-overridable step.
Quality: ThreadLocal used to pass Include through subclass overrides

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:363-364 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1914-1923 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:8499-8500
The bulkInclude ThreadLocal (line 363) is used as an implicit parameter to pass Include through the 2-param setFieldsInBulkfetchAndSetFields call chain, because the 2-param setFieldsInBulk is overridden by 21+ subclasses and cannot have its signature changed.

While the save/restore in the finally block (lines 1916-1922) is correct, ThreadLocal-based implicit parameter passing is fragile:

  1. Any future code that calls the 2-param fetchAndSetFields(entities, fields) directly (e.g., from a subclass override) will silently read whatever value happens to be on the ThreadLocal, defaulting to NON_DELETED which is safe but could mask bugs.
  2. The coupling is invisible — a developer reading a subclass override has no indication that Include semantics flow through a ThreadLocal.

Consider adding a // NOTE: Include is passed via bulkInclude ThreadLocal comment on the 2-param setFieldsInBulk and fetchAndSetFields methods, or documenting this in the class-level Javadoc so future maintainers are aware of the pattern.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:2019

  • listInternal(...) now accepts an Include include argument but never uses it. This adds dead code / confusion and suggests the include-threading refactor wasn't fully cleaned up. Either remove the include parameter (and revert call sites), or use it for actual behavior if needed.
  private List<T> listInternal(
      List<String> jsons, Fields fields, UriInfo uriInfo, Include include) {
    List<T> entities;
    try (var ignored = phase("jsonDeserialize")) {
      entities = JsonUtils.readObjects(jsons, entityClass);
    }
    try (var ignored = phase("setFieldsBulk")) {
      setFieldsInBulk(fields, entities);
    }

Comment on lines +1313 to +1314
ListParams params = new ListParams().setFields("followers").withLimit(1000000);
ListResponse<Domain> list = listEntities(params);
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test uses withLimit(1000000) to ensure the target domain appears in the list. This can make the test slower/flakier as the suite grows and can stress the list endpoint unnecessarily. Prefer paging (like the earlier experts tests in this file) or a smaller limit combined with cursor iteration.

Copilot uses AI. Check for mistakes.
Comment on lines +1347 to +1348
ListParams params = new ListParams().setFields("votes").withLimit(1000000);
ListResponse<Domain> list = listEntities(params);
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test uses withLimit(1000000) to fetch votes in one call. To keep integration tests performant and robust, prefer paginating until the entity is found (or use a smaller limit) rather than requesting an extremely large page size.

Copilot uses AI. Check for mistakes.

private Iterator<Either<T, EntityError>> serializeJsons(
List<String> jsons, Fields fields, UriInfo uriInfo) {
List<String> jsons, Fields fields, UriInfo uriInfo, Include include) {
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serializeJsons(...) was updated to take an Include include parameter, but the parameter is not referenced anywhere in the method. This looks like leftover plumbing from the removed include-threading approach; consider removing the unused parameter and updating callers to avoid misleading APIs.

Suggested change
List<String> jsons, Fields fields, UriInfo uriInfo, Include include) {
List<String> jsons, Fields fields, UriInfo uriInfo) {

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Soft deleted users still appear in Experts/Reviewers field in Data Product

4 participants