Grants version going backward during write workload

[pingtimeout]

Describe the bug

While I was testing the apache/polaris-tools#173 code, I ran into lot of errors like the one below during the write workload:

2026-02-19 11:29:43,998 ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [a4ef00c5-6eba-4c69-8e57-f35c13548212_0000000000000000676,POLARIS] [,,,] (executor-thread-6) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.IllegalStateException: grants_version_going_backward: entity={} grantsVersion={}, [name=catalog_administrator_C_0;id=660631469561579369;parentId=1137621942110885014;entityVersion=1;type=CATALOG_ROLE;subType=NULL_SUBTYPE;internalProperties={}, 136]
        at com.google.common.base.Preconditions.checkState(Preconditions.java:853)
        at org.apache.polaris.core.PolarisDefaultDiagServiceImpl.check(PolarisDefaultDiagServiceImpl.java:126)
        at org.apache.polaris.core.persistence.ResolvedPolarisEntity.<init>(ResolvedPolarisEntity.java:66)
        at org.apache.polaris.core.persistence.cache.InMemoryEntityCache.getAndRefreshIfNeeded(InMemoryEntityCache.java:343)
        at org.apache.polaris.core.persistence.resolver.Resolver.bulkValidate(Resolver.java:576)
        at org.apache.polaris.core.persistence.resolver.Resolver.runResolvePass(Resolver.java:440)
        at org.apache.polaris.core.persistence.resolver.Resolver.resolveAll(Resolver.java:250)
        at org.apache.polaris.core.persistence.resolver.PolarisResolutionManifest.resolveAll(PolarisResolutionManifest.java:135)
        at org.apache.polaris.service.admin.PolarisAdminService.authorizeGrantOnNamespaceOperationOrThrow(PolarisAdminService.java:459)
        at org.apache.polaris.service.admin.PolarisAdminService.grantPrivilegeOnNamespaceToRole(PolarisAdminService.java:1690)
        at org.apache.polaris.service.admin.PolarisAdminService_ClientProxy.grantPrivilegeOnNamespaceToRole(Unknown Source)
        at org.apache.polaris.service.admin.PolarisServiceImpl.addGrantToCatalogRole(PolarisServiceImpl.java:641)
        at org.apache.polaris.service.admin.PolarisServiceImpl_Subclass.addGrantToCatalogRole$$superforward(Unknown Source)
        at org.apache.polaris.service.admin.PolarisCatalogsEventServiceDelegator_BLUoP2dn7JG2IbxCE7Nu_CXmT7s_Delegate_Subclass.addGrantToCatalogRole(Unknown Source)
        at org.apache.polaris.service.admin.PolarisCatalogsEventServiceDelegator.addGrantToCatalogRole(PolarisCatalogsEventServiceDelegator.java:199)
        at org.apache.polaris.service.admin.PolarisServiceImpl_Subclass.addGrantToCatalogRole(Unknown Source)
        at org.apache.polaris.service.admin.PolarisServiceImpl_ClientProxy.addGrantToCatalogRole(Unknown Source)
        at org.apache.polaris.service.admin.api.PolarisCatalogsApi.addGrantToCatalogRole(PolarisCatalogsApi.java:105)
        at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass.addGrantToCatalogRole$$superforward(Unknown Source)
        at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass$$function$$1.apply(Unknown Source)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.quarkus.hibernate.validator.runtime.interceptor.AbstractMethodValidationInterceptor.validateMethodInvocation(AbstractMethodValidationInterceptor.java:71)
        at io.quarkus.hibernate.validator.runtime.jaxrs.ResteasyReactiveEndPointValidationInterceptor.validateMethodInvocation(ResteasyReactiveEndPointValidationInterceptor.java:21)
        at io.quarkus.hibernate.validator.runtime.jaxrs.ResteasyReactiveEndPointValidationInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364)
        at io.smallrye.faulttolerance.core.Future.from(Future.java:85)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364)
        at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20)
        at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)
        at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
        at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)
        at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29)
        at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
        at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:44)
        at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor_RolesAllowedInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
        at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
        at org.apache.polaris.service.admin.api.PolarisCatalogsApi_Subclass.addGrantToCatalogRole(Unknown Source)
        at org.apache.polaris.service.admin.api.PolarisCatalogsApi$quarkusrestinvoker$addGrantToCatalogRole_56882127262eb1301ad4a14f5e3fb19c9e966db3.invoke(Unknown Source)
        at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
        at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:183)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)

This caused multiple HTTP 500 errors on the client side. Attached is the full Polaris log file. polaris.log.gz

To Reproduce

1. Checkout the branch from Add RBAC support to write benchmark polaris-tools#173 (until it is merged).
2. Create an application.conf file as given below, adjust the Polaris base URL accordingly
3. Run make create-dataset-simulation from the benchmarks/ directory

http {
  base-url = "http://localhost:8181"
}
 
auth {
  client-id = "root"
  client-secret = "s3cr3t"
}
 
rbac {
  enabled = true
  num-principals = 5
  catalog-role-names = ["catalog_reader", "catalog_contributor"]
  principal-role-names = ["data_engineer", "data_scientist"]
}
 
dataset.tree {
  num-catalogs = 5
 
  namespace-width = 10
  namespace-depth = 5
  tables-per-namespace = 10
  views-per-namespace = 10
 
  columns-per-table = 10
  columns-per-view = 10
 
  default-base-location = "file:///tmp/polaris"
 
  namespace-properties = 10
  table-properties = 10
  view-properties = 10
}
 
workload {
  create-tree-dataset {
    table-concurrency = 10
    view-concurrency = 10
  }
}

Actual Behavior

No response

Expected Behavior

No response

Additional context

No response

System information

No response

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment if you think it's still relevant.