Skip to content

refactor: extract stateless-db crate and split pipeline/validator modules #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
flyq wants to merge 10 commits into
base: main
Choose a base branch
from
Open

refactor: extract stateless-db crate and split pipeline/validator modules #122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
27ae7b5
refactor4
flyq Apr 21, 2026
f7ed7db
opt
flyq Apr 21, 2026
964670f
opt
flyq Apr 22, 2026
fb78b77
fix ci & comments
flyq Apr 22, 2026
a67bb16
fix ci
flyq Apr 22, 2026
256b97a
fix claude review comments
flyq Apr 22, 2026
9b3129c
unify the retry & backoff of RPCs
flyq Apr 22, 2026
957f87a
opt: debug-trace-server hot-path
flyq Apr 22, 2026
dc41039
simplify
flyq Apr 22, 2026
5fd690a
update inline comments & merge the backoff
flyq Apr 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 28 additions & 28 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,14 @@ The project uses nightly `2026-02-03` toolchain (edition 2024, rust-version 1.95

## Workspace Structure

| Crate | Path | Purpose |
| ---------------------- | ----------------------------- | ---------------------------------------------------------- |
| `stateless-core` | `crates/stateless-core` | Core validation logic, database, EVM execution, RPC client |
| `stateless-common` | `crates/stateless-common` | Shared infra: RpcClient, metrics, logging, redb table defs |
| `stateless-test-utils` | `crates/stateless-test-utils` | Test fixtures (blocks, witnesses, contracts) + env lock |
| `stateless-validator` | `bin/stateless-validator` | Main binary: chain sync, parallel validation workers |
| `debug-trace-server` | `bin/debug-trace-server` | Standalone RPC server for debug/trace methods |
| Crate | Path | Purpose |
| ---------------------- | ----------------------------- | ------------------------------------------------------------------------------------------ |
| `stateless-core` | `crates/stateless-core` | Storage traits, pipeline, EVM execution, SALT witness handling, chain spec, error types |
| `stateless-db` | `crates/stateless-db` | redb-backed persistence: table definitions, read/write helpers, `ContractCache` |
| `stateless-common` | `crates/stateless-common` | RPC client, metrics/logging utilities, witness size estimation |
| `stateless-test-utils` | `crates/stateless-test-utils` | Test fixtures (blocks, witnesses, contracts) and env-var lock for integration tests |
| `stateless-validator` | `bin/stateless-validator` | Main binary: chain sync, parallel validation workers (`app.rs` / `workers.rs` / `main.rs`) |
| `debug-trace-server` | `bin/debug-trace-server` | Standalone RPC server for debug/trace methods |

Comment thread
flyq marked this conversation as resolved.
Additional directories: `test_data/` (integration test fixtures including genesis config), `audits/` (security audit reports).

Expand All @@ -58,12 +59,13 @@ The outer loop (`run_pipeline`) handles reorg rollback + restart, stale-data anc

### Database

The validator and trace server each have their own `redb`-backed database.
Shared table definitions live in `stateless-common::db`:
The validator and trace server each own a `redb`-backed database (`ValidatorDB` and `ServerDB`, living in their respective binaries).
The shared persistence layer — table definitions, read/write helpers, serialization, and `ContractCache` — lives in the `stateless-db` crate.
`stateless-core::db` defines the abstract storage traits (`ChainStore`, `BlockStore`, etc.) and the `StoreError` / `StoreResult` returned by them.

- **`ANCHOR_BLOCK`** — Trusted starting point (block number, hash, state root, withdrawals root).
- **`CANONICAL_CHAIN`** — Validated chain progression (block number → hash, state root, withdrawals root).
- **`CONTRACTS`** — On-demand contract bytecode cache (code hash → bincode+lz4 bytecode).
- **`CONTRACTS`** — Persistent tier of the contract bytecode cache (code hash → bincode+lz4 bytecode). The in-memory tier is the bounded `ContractCache` on top.
- **`GENESIS_CONFIG`** — Hardfork activation rules (validator only).
- **`BLOCK_DATA`** — Full block content (trace server only).
- **`WITNESSES`** — Light witness data (trace server only).
Expand Down Expand Up @@ -104,28 +106,26 @@ The server includes an HTTP response cache (`quick_cache`) for pre-serialized JS

### Key Source Files

| File | Purpose |
| --------------------------------------------- | ----------------------------------------------------------------------------------- |
| `crates/stateless-core/src/pipeline.rs` | Generic pipeline: BlockFetcher, run_pipeline, chain_advancer, find_divergence_point |
| `crates/stateless-core/src/executor.rs` | Block validation and EVM replay |
| `crates/stateless-core/src/evm_database.rs` | WitnessDatabase implementing `revm::DatabaseRef` |
| `crates/stateless-core/src/db.rs` | Abstract storage traits (ChainStore, BlockStore, etc.) |
| `crates/stateless-core/src/withdrawals.rs` | Withdrawal validation and MPT witness handling |
| `crates/stateless-common/src/rpc_client.rs` | RPC client for blocks, witnesses, and bytecode |
| `crates/stateless-common/src/db.rs` | Shared redb table definitions and serialization helpers |
| `crates/stateless-common/src/metrics.rs` | RpcMethod, RpcMetrics, RpcClientConfig |
| `crates/stateless-common/src/witness_size.rs` | `WitnessSizeBreakdown` + `estimate_witness_size` for RPC and trace-server metrics |
| `crates/stateless-test-utils/src/fixtures.rs` | `TestFixtures` loader (blocks, SALT/MPT witnesses, contracts, genesis) |
| `bin/stateless-validator/src/chain_sync.rs` | ValidatorFetcher, ValidatorProcessor, ValidatorHooks |
| `bin/stateless-validator/src/main.rs` | CLI, pipeline startup, validation reporter |
| `bin/debug-trace-server/src/chain_sync.rs` | TraceFetcher, TraceProcessor, TraceHooks |
| `bin/debug-trace-server/src/rpc_service.rs` | RPC method definitions and handlers |
| `bin/debug-trace-server/src/data_provider.rs` | Block data fetching with single-flight coalescing |
| File | Purpose |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `crates/stateless-core/src/pipeline/{mod,config,traits,fetcher,divergence,advancer,worker}.rs` | Generic three-stage pipeline split by responsibility |
| `crates/stateless-core/src/executor.rs` | Block validation and EVM replay |
| `crates/stateless-core/src/evm_database.rs` | WitnessDatabase implementing `revm::DatabaseRef` |
| `crates/stateless-core/src/db.rs` | Abstract storage traits + `StoreError` / `StoreResult` |
| `crates/stateless-core/src/withdrawals.rs` | Withdrawal validation and MPT witness handling |
| `crates/stateless-db/src/{lib,tables,helpers,serialize,cache}.rs` | Shared redb tables, helpers, serialization, and `ContractCache` |
| `crates/stateless-common/src/rpc_client.rs` | RPC client for blocks, witnesses, and bytecode |
| `crates/stateless-common/src/metrics.rs` | RpcMethod, RpcMetrics, RpcClientConfig |
| `bin/stateless-validator/src/{main,app,workers,chain_sync,validator_db,metrics}.rs` | Thin entry, CLI/startup wiring, pipeline+reporter, fetcher/processor, DB |
| `bin/debug-trace-server/src/chain_sync.rs` | TraceFetcher, TraceProcessor, TraceHooks |
| `bin/debug-trace-server/src/rpc_service.rs` | RPC method definitions and handlers |
| `bin/debug-trace-server/src/data_provider.rs` | Block data fetching with single-flight coalescing |
| `bin/debug-trace-server/src/server_db.rs` | Concrete `BlockStore` implementation backed by `stateless-db` |

## Test Organization

Unit tests are embedded in source files alongside the code they test.
Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, consistency, performance, timing_header, prune).
Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, consistency, performance, timing_header, prune) and in `bin/stateless-validator/tests/integration.rs` (CLI parsing, mock-RPC pipeline, mainnet single-block validation).
Test data (block JSON files, contract bytecode, witness data) is stored in `test_data/`.

## Version Control
Expand Down
37 changes: 30 additions & 7 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ members = [
"bin/stateless-validator",
"crates/stateless-common",
"crates/stateless-core",
"crates/stateless-db",
"crates/stateless-test-utils",
]
resolver = "2"
Expand Down Expand Up @@ -96,7 +97,7 @@ tokio-util = { version = "0.7", default-features = false }
tower = "0.5"
tracing = { version = "0.1", default-features = false }
tracing-appender = "0.2"
tracing-subscriber = { version = "=0.3.23", features = ["fmt", "env-filter", "json"], default-features = false }
tracing-subscriber = { version = "=0.3.23", features = ["ansi", "env-filter", "fmt", "json"], default-features = false }
zstd = { version = "0.13", default-features = false }

[profile.release]
Expand Down
51 changes: 27 additions & 24 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,16 @@ The stateless approach eliminates the need for validators to run on high-end har

## Project Structure

The workspace contains two binaries and two library crates:
The workspace contains two binaries and four library crates:

| Crate | Path | Purpose |
| --------------------- | ------------------------- | ---------------------------------------------------------------- |
| `stateless-core` | `crates/stateless-core` | Core validation logic, abstract traits, generic pipeline |
| `stateless-common` | `crates/stateless-common` | Shared utilities: RPC client, database helpers, logging, metrics |
| `stateless-validator` | `bin/stateless-validator` | Main binary: chain sync, parallel validation workers |
| `debug-trace-server` | `bin/debug-trace-server` | Standalone RPC server for debug/trace methods |
| Crate | Path | Purpose |
| ---------------------- | ----------------------------- | ----------------------------------------------------------------------------------- |
| `stateless-core` | `crates/stateless-core` | Core validation logic, abstract storage traits, generic pipeline, EVM execution |
| `stateless-db` | `crates/stateless-db` | redb-backed persistence: table definitions, read/write helpers, bounded `ContractCache` |
| `stateless-common` | `crates/stateless-common` | Shared utilities: RPC client, logging, metrics |
| `stateless-test-utils` | `crates/stateless-test-utils` | Test fixtures (blocks, witnesses, contracts) and env-var lock for integration tests |
| `stateless-validator` | `bin/stateless-validator` | Main binary: chain sync, parallel validation workers |
| `debug-trace-server` | `bin/debug-trace-server` | Standalone RPC server for debug/trace methods |

Additional directories: `test_data/` (integration test fixtures including genesis config), `audits/` (security audit reports).

Expand Down Expand Up @@ -194,26 +196,27 @@ The pipeline is configured via `PipelineConfig` and customized through trait imp

### Key Source Files

| File | Purpose |
| --------------------------------------------- | ----------------------------------------------------------------------------------- |
| `crates/stateless-core/src/pipeline.rs` | Generic pipeline: BlockFetcher, run_pipeline, chain_advancer, find_divergence_point |
| `crates/stateless-core/src/executor.rs` | Block validation and EVM replay |
| `crates/stateless-core/src/db.rs` | Storage traits: ChainStore, BlockStore, ContractStore |
| `crates/stateless-core/src/evm_database.rs` | WitnessDatabase implementing `revm::DatabaseRef` |
| `crates/stateless-core/src/withdrawals.rs` | Withdrawal validation and MPT witness handling |
| `crates/stateless-common/src/rpc_client.rs` | RpcClient: HTTP-based block/witness/contract fetching |
| `crates/stateless-common/src/db.rs` | Shared redb table definitions and helpers |
| `crates/stateless-common/src/metrics.rs` | RpcMethod, RpcMetrics, RpcClientConfig |
| `crates/stateless-common/src/witness_size.rs` | `WitnessSizeBreakdown` + `estimate_witness_size` for RPC and trace-server metrics |
| `crates/stateless-test-utils/src/fixtures.rs` | `TestFixtures` loader (blocks, SALT/MPT witnesses, contracts, genesis) |
| `bin/stateless-validator/src/chain_sync.rs` | ValidatorFetcher, ValidatorProcessor, ValidatorHooks |
| `bin/debug-trace-server/src/chain_sync.rs` | TraceFetcher, TraceProcessor, TraceHooks |
| `bin/debug-trace-server/src/rpc_service.rs` | RPC method definitions and handlers |
| `bin/debug-trace-server/src/data_provider.rs` | Block data fetching with single-flight coalescing |
| File | Purpose |
| ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `crates/stateless-core/src/pipeline/{mod,config,traits,fetcher,divergence,advancer,worker}.rs` | Generic three-stage pipeline split by responsibility |
| `crates/stateless-core/src/executor.rs` | Block validation and EVM replay |
| `crates/stateless-core/src/db.rs` | Abstract storage traits: `ChainStore`, `BlockStore`, `ContractStore`, `StoreError` |
| `crates/stateless-core/src/evm_database.rs` | `WitnessDatabase` implementing `revm::DatabaseRef` |
| `crates/stateless-core/src/withdrawals.rs` | Withdrawal validation and MPT witness handling |
| `crates/stateless-db/src/{lib,tables,helpers,serialize,cache}.rs` | Shared redb tables, helpers, serialization, and bounded `ContractCache` |
| `crates/stateless-common/src/rpc_client.rs` | `RpcClient`: multi-endpoint HTTP client for blocks, witnesses, and bytecode |
| `crates/stateless-common/src/metrics.rs` | `RpcMethod`, `RpcMetrics`, `RpcClientConfig` |
| `crates/stateless-common/src/witness_size.rs` | `WitnessSizeBreakdown` + `estimate_witness_size` for RPC and trace-server metrics |
| `crates/stateless-test-utils/src/fixtures.rs` | `TestFixtures` loader (blocks, SALT/MPT witnesses, contracts, genesis) |
| `bin/stateless-validator/src/{main,app,workers,chain_sync,validator_db,metrics}.rs` | Thin entry, CLI/startup wiring, pipeline+reporter, fetcher/processor, DB |
| `bin/debug-trace-server/src/chain_sync.rs` | `TraceFetcher`, `TraceProcessor`, `TraceHooks` |
| `bin/debug-trace-server/src/rpc_service.rs` | RPC method definitions and handlers |
| `bin/debug-trace-server/src/data_provider.rs` | Block data fetching with single-flight coalescing |
| `bin/debug-trace-server/src/server_db.rs` | Concrete `BlockStore` implementation backed by `stateless-db` |

### Database

The validator and trace server each have their own `redb`-backed database, sharing common table definitions from `stateless-common::db`:
The validator and trace server each have their own `redb`-backed database, sharing common table definitions and helpers from the `stateless-db` crate:

| Table | Key | Value | Used By |
| ----------------- | -------------------------- | ------------------------------------------------------ | ---------------------------- |
Expand Down
Loading
Loading