Skip to content

Restarting DM validator won't bring the metrics back #12601

@kennytm

Description

@kennytm

What did you do?

  1. Start a task without validator.
  2. Start validator.
tiup dmctl:v8.5.5 --master-addr=127.0.0.1:8261 validation start --mode full t1
  1. Run for a while, verify that "Validator Statistics" in Grafana is populated. (Equivalently, the Prometheus metric dm_validator_validator_binlog_pos is present).

  2. Stop and restart validator

tiup dmctl:v8.5.5 --master-addr=127.0.0.1:8261 validation stop t1
tiup dmctl:v8.5.5 --master-addr=127.0.0.1:8261 validation start t1
  1. Verify via dmctl query-status that the validator is still running
  2. Check Grafana again

What did you expect to see?

Metrics present

What did you see instead?

No metrics

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

v8.5.5

Upstream MySQL/MariaDB server version:

v5.7

Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

v8.5.5

How did you deploy DM: tiup or manually?

tiup playground

current status of DM cluster (execute query-status <task-name> in dmctl)

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-8.5This bug affects the 8.5.x(LTS) versions.area/dmIssues or PRs related to DM.found/gstype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions