[Bug] PROPERTY-SCHEMA-REGISTRY failed to re-connect to gRPC server on port 17916 in standalone mod

[ardacuhadaroglu]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

BanyanDB (apache/skywalking-banyandb)

What happened

I am using the official Apache SkyWalking Helm Chart (4.8.0) with the default configuration. The only modifications made to the values.yaml are the image tags (OAP: 10.4.0, BanyanDB: 0.10.1).
The SkyWalking banyan db server is reporting continuous gRPC connection failures when attempting to connect to BanyanDB schema registry on port 17916.
In the logs, the PROPERTY-SCHEMA-REGISTRY module identifies the node skywalking-banyandb-0:17912 as unhealthy and moves it to the evict queue. Subsequently, it tries to reconnect to the property_schema_grpc_address at skywalking-banyandb-0:17916, but fails repeatedly with the following error: failed to re-connect to grpc server skywalking-banyandb-0:17916 after waiting for [X]s.
Even though BanyanDB is running in standalone mode (default in the chart), the OAP server seems to be attempting to reach a specific gRPC port (17916) for property schema management which appears to be unreachable.

What you expected to happen

Using the official Helm chart out-of-the-box with updated image versions should result in a stable connection. The PROPERTY-SCHEMA-REGISTRY should successfully register and communicate with the BanyanDB node without being marked as unhealthy, and it should use the correct, accessible gRPC ports provided by the standalone deployment.

How to reproduce

1. Use the official SkyWalking Helm chart (unmodified, except for image tags).
2. Set oap.image.tag to 10.4.0 and banyandb.image.tag to 0.10.1.
3. Deploy the chart with BanyanDB enabled in standalone mode (default: banyandb.standalone.enabled=true).
4. Observe the banyan db logs shortly after startup.
5. Notice the node being evicted from the register flow and the subsequent gRPC reconnection failures to port 17916.

Anything else

Banyan db logs attached.

skywalking-banyandb-0_standalone.log

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hanahmily]

Hi @ardacuhadaroglu, thanks for the detailed report — confirmed this is a chart/image version mismatch.

Root cause

The PROPERTY-SCHEMA-REGISTRY on port 17916 is a new component in BanyanDB, introduced via apache/skywalking-banyandb#971 (part of #13614). The embedded schema server defaults to --schema-server-grpc-port=17916, and OAP 10.4.0 expects to reach it there.

SkyWalking Helm 4.8.0 still depends on skywalking-banyandb-helm: 0.5.2, which predates this server — so no 17916 port/Service is provisioned and no --schema-server-* flags are passed to banyand. That's why OAP's property-schema-registry keeps failing to reach skywalking-banyandb-0:17916.

The dependency bumps to 0.6.0-rc4 (PR #178) and 0.6.0-rc5 (PR #179) have already landed on master, but aren't in a released chart yet.

Suggested fix: install the snapshot chart from master

Until the next official release, please try the snapshot published to ghcr.io, which already pulls in banyandb-helm 0.6.0-rc5:

helm install skywalking \
  oci://ghcr.io/apache/skywalking-helm/skywalking-helm \
  --version 0.0.0-<full-commit-sha-of-master> \
  -n <your-namespace> \
  --set oap.image.tag=10.4.0 \
  --set oap.storageType=banyandb \
  --set ui.image.tag=10.4.0 \
  --set elasticsearch.enabled=false \
  --set banyandb.enabled=true \
  --set banyandb.image.tag=0.10.1

Use any commit SHA on master that's at or after PR #179 (ef1ca5b). This should expose 17916 correctly and let the property-schema-registry connect. Please let us know if it resolves the reconnect loop on your side.

/cc @mrproliu

[ardacuhadaroglu]

Thanks, it worked. I'll use this version in a production, hopefully everything goes smoothly.

[ardacuhadaroglu]

@hanahmily I am getting same error on banyandb again. What can i do? I used exactly this command to install. I don't always get this error, but sometimes when I reinstall it, I start getting it again. It's very strange. I tried it when you commented and it worked, but now it doesn't.

helm install skywalking
oci://ghcr.io/apache/skywalking-helm/skywalking-helm
--version 0.0.0-e891f81f40cfda509cca823a6214beac8e83ecb7
-n cloud-infra
--set oap.image.tag=10.4.0
--set oap.storageType=banyandb
--set ui.image.tag=10.4.0
--set elasticsearch.enabled=false
--set banyandb.enabled=true
--set banyandb.image.tag=0.10.1

Logs:

{"level":"info","module":"PROPERTY-SCHEMA-REGISTRY","node":"skywalking-banyandb-0:17912","time":"2026-04-18T14:49:25Z","message":"schema server node is inactive"}
{"level":"info","module":"PROPERTY-SCHEMA-REGISTRY","status":"registered: [skywalking-banyandb-0:17912], active :[], evictable :[skywalking-banyandb-0:17912]","node":"metadata:{name:"skywalking-banyandb-0:17912"} roles:ROLE_META roles:ROLE_DATA roles:ROLE_LIAISON grpc_address:"skywalking-banyandb-0:17912" property_schema_grpc_address:"skywalking-banyandb-0:17916"","time":"2026-04-18T14:49:25Z","message":"node is unhealthy in the register flow, move it to evict queue"}
{"level":"error","module":"PROPERTY-SCHEMA-REGISTRY","time":"2026-04-18T14:49:31Z","message":"failed to re-connect to grpc server skywalking-banyandb-0:17916 after waiting for 1.078668361s"}
{"level":"error","module":"PROPERTY-SCHEMA-REGISTRY","time":"2026-04-18T14:49:38Z","message":"failed to re-connect to grpc server skywalking-banyandb-0:17916 after waiting for 2.143659122s"}
{"level":"error","module":"PROPERTY-SCHEMA-REGISTRY","time":"2026-04-18T14:49:47Z","message":"failed to re-connect to grpc server skywalking-banyandb-0:17916 after waiting for 3.905605413s"}
{"level":"error","module":"PROPERTY-SCHEMA-REGISTRY","time":"2026-04-18T14:49:59Z","message":"failed to re-connect to grpc server skywalking-banyandb-0:17916 after waiting for 7.091133235s"}
{"level":"error","module":"PROPERTY-SCHEMA-REGISTRY","time":"2026-04-18T14:50:18Z","message":"failed to re-connect to grpc server skywalking-banyandb-0:17916 after waiting for 13.515657952s"}
{"level":"warn","module":"METADATA","attempt":3,"error":"no schema servers reachable after 1m0s","time":"2026-04-18T14:50:25Z","message":"failed to create property schema registry, retrying..."}

[wu-sheng]

Could you share how to reproduce it? skywalking.apache.org#demo is using this installation helm, it seems good. And we updated and rebooted the env from time to time. Your sometimes should have some specific conditions.

[ardacuhadaroglu]

I deleted all images on my local machine, cleaned all storage and pods via rancher-desktop, and created a clean environment. Then I ran Helm Install. Now I get this error every time.

Meanwhile, I encountered the same error on Docker. Adding this to the command section fixed the problem.

--node-host-provider=ip

[wu-sheng]

I can't reproduce this, as I am using local Kind/Docker-Desktop. Do you have a real K8s to recheck? Our GCP K8s is good for this helm.

[hanahmily]

The logs actually tell a different story from the first report — port 17916 is now wired in (that's why it sometimes works with the snapshot chart), so this looks like a DNS / Service discovery issue rather than a missing port.

Key evidence from your logs:

• Standalone node advertises itself as property_schema_grpc_address: "skywalking-banyandb-0:17916" — a bare pod hostname, not a FQDN or IP.
• BanyanDB uses DNS-based service discovery to reach the schema server, so that short name must resolve inside the cluster.
• In a StatefulSet, skywalking-banyandb-0 only gets a DNS A record via its headless Service, and typically only once the pod is Ready — unless the Service sets publishNotReadyAddresses: true. During the registration flow the pod is still NotReady, so the lookup the node needs in order to register can fail. Whether CoreDNS already has the record cached from a prior run is what makes this intermittent across reinstalls.

Could you run the following on a failing install and share the output? This will tell us exactly where discovery is breaking:

# 1. Does the headless Service expose 17916 and publish NotReady addresses?
kubectl -n cloud-infra get svc -l app.kubernetes.io/name=banyandb -o yaml \
  | grep -E "name:|port:|targetPort:|publishNotReadyAddresses"

# 2. Is the schema server actually listening on 17916 inside the pod?
kubectl -n cloud-infra exec skywalking-banyandb-0 -- \
  sh -c 'ss -tlnp 2>/dev/null || netstat -tlnp'

# 3. Does the advertised hostname resolve from inside the pod?
kubectl -n cloud-infra exec skywalking-banyandb-0 -- nslookup skywalking-banyandb-0
kubectl -n cloud-infra exec skywalking-banyandb-0 -- \
  nslookup skywalking-banyandb-0.skywalking-banyandb-grpc.cloud-infra.svc.cluster.local

# 4. Can we reach 17916 on that name from the pod itself?
kubectl -n cloud-infra exec skywalking-banyandb-0 -- \
  nc -zv skywalking-banyandb-0 17916

My expectation is that (1) will show the standalone headless Service missing publishNotReadyAddresses: true (or port 17916 not listed), and (3) will fail for the short name while the FQDN resolves. If that's the case, the fix is on the chart side — either expose 17916 on a headless Service with publishNotReadyAddresses: true, or have the standalone node advertise itself via IP (--node-host-provider=ip) so registration doesn't depend on DNS readiness at all.