test(dm): kill dm-masters sequentially in cleanup_process

In multi-master HA tests (3-node etcd cluster), sending SIGHUP to
all masters simultaneously causes etcd to lose quorum — each master
tries to transfer leadership but no peer can accept it. The leader
transfer blocks for 120s, failing the test.

Fix: kill dm-masters one at a time (SIGHUP + 30s wait per master),
so each graceful shutdown completes while quorum is maintained.
Escalate to SIGKILL after 30s for any stuck master. Workers and
syncers are still killed in parallel (no quorum dependency).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: joechenrh

dm/tests/_utils/test_prepare

@@ -18,15 +18,27 @@ function cleanup_data_upstream() {
 }
 
 function cleanup_process() {
-    dm_master_num=$(ps aux >temp && grep "dm-master.test" temp | wc -l && rm temp)
-    echo "$dm_master_num dm-master alive"
-    pkill -hup dm-master.test 2>/dev/null || true
+    # Kill dm-masters one at a time to maintain etcd quorum during graceful
+    # shutdown. Killing all 3 simultaneously causes etcd to lose quorum,
+    # blocking leader transfer indefinitely.
+    local pids
+    pids=$(pgrep -f dm-master.test || true)
+    echo "$(echo "$pids" | wc -w) dm-master alive"
+    for pid in $pids; do
+        kill -HUP $pid 2>/dev/null || true
+        for _ in $(seq 1 30); do
+            if ! kill -0 $pid 2>/dev/null; then break; fi
+            sleep 1
+        done
+        # Escalate if still alive after 30s
+        kill -9 $pid 2>/dev/null || true
+    done
 
-    dm_worker_num=$(ps aux >temp && grep "dm-worker.test" temp | wc -l && rm temp)
+    dm_worker_num=$(pgrep -c -f dm-worker.test || true)
     echo "$dm_worker_num dm-worker alive"
     pkill -hup dm-worker.test 2>/dev/null || true
 
-    dm_syncer_num=$(ps aux >temp && grep "dm-syncer.test" temp | wc -l && rm temp)
+    dm_syncer_num=$(pgrep -c -f dm-syncer.test || true)
     echo "$dm_syncer_num dm-syncer alive"
     pkill -hup dm-syncer.test 2>/dev/null || true