Skip to content

Commit 53bd6c1

Browse files
abrichrclaude
andcommitted
Add architecture diagrams and updated permissions guide
- docs/architecture.md: Mermaid diagrams for system overview, data flow, package dependencies, and evaluation loop - docs/permissions-macos.md: Updated macOS permissions guide for 13-15 with troubleshooting - README.md: Updated with Mermaid flow diagram Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 33e7998 commit 53bd6c1

File tree

3 files changed

+467
-3
lines changed

3 files changed

+467
-3
lines changed

README.md

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -96,9 +96,33 @@ openadapt doctor Check system requirements
9696

9797
## How It Works
9898

99-
<img width="1499" alt="OpenAdapt Architecture" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/c811654e-3450-42cd-91ee-935378e3a858">
100-
101-
<img width="1511" alt="OpenAdapt Flow" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/82814cdb-f0d5-4a6b-9d44-a4628fca1590">
99+
See the full [Architecture Documentation](docs/architecture.md) for detailed diagrams.
100+
101+
```mermaid
102+
flowchart LR
103+
subgraph Record["1. Record"]
104+
A[User Demo] --> B[Capture]
105+
end
106+
107+
subgraph Train["2. Train"]
108+
B --> C[ML Model]
109+
end
110+
111+
subgraph Deploy["3. Deploy"]
112+
C --> D[Agent Policy]
113+
D --> E[Action Replay]
114+
end
115+
116+
subgraph Evaluate["4. Evaluate"]
117+
D --> F[Benchmark]
118+
F --> G[Metrics]
119+
end
120+
121+
%% Optional enhancements
122+
GROUND[Grounding] -.-> E
123+
RETRIEVE[Retrieval] -.-> C
124+
PRIV[Privacy] -.-> B
125+
```
102126

103127
OpenAdapt:
104128
- Records screenshots and user input events

docs/architecture.md

Lines changed: 295 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,295 @@
1+
# OpenAdapt Architecture
2+
3+
OpenAdapt v1.0+ uses a **modular meta-package architecture** where the main `openadapt` package provides a unified CLI and depends on focused sub-packages.
4+
5+
## System Overview
6+
7+
```mermaid
8+
flowchart TB
9+
subgraph User["User"]
10+
UI[Desktop/Web GUI]
11+
end
12+
13+
subgraph OpenAdapt["OpenAdapt Meta-Package"]
14+
CLI[openadapt CLI]
15+
LAZY[Lazy Imports]
16+
end
17+
18+
subgraph Core["Core Packages"]
19+
CAPTURE[openadapt-capture]
20+
ML[openadapt-ml]
21+
EVALS[openadapt-evals]
22+
VIEWER[openadapt-viewer]
23+
end
24+
25+
subgraph Optional["Optional Packages"]
26+
GROUNDING[openadapt-grounding]
27+
RETRIEVAL[openadapt-retrieval]
28+
PRIVACY[openadapt-privacy]
29+
end
30+
31+
subgraph Storage["Storage"]
32+
DEMO[(Demonstration<br/>JSON/Parquet)]
33+
MODEL[(Model<br/>Checkpoints)]
34+
RESULTS[(Evaluation<br/>Results)]
35+
end
36+
37+
%% User interactions
38+
UI --> CAPTURE
39+
40+
%% CLI orchestration
41+
CLI --> CAPTURE
42+
CLI --> ML
43+
CLI --> EVALS
44+
CLI --> VIEWER
45+
46+
%% Lazy loading
47+
LAZY -.-> GROUNDING
48+
LAZY -.-> RETRIEVAL
49+
LAZY -.-> PRIVACY
50+
51+
%% Data flow
52+
CAPTURE --> DEMO
53+
DEMO --> ML
54+
ML --> MODEL
55+
MODEL --> EVALS
56+
EVALS --> RESULTS
57+
DEMO --> VIEWER
58+
59+
%% Optional integrations
60+
GROUNDING -.-> ML
61+
RETRIEVAL -.-> ML
62+
PRIVACY -.-> CAPTURE
63+
PRIVACY -.-> VIEWER
64+
65+
classDef metaPkg fill:#4A90D9,stroke:#2E5A8B,color:#fff
66+
classDef corePkg fill:#5CB85C,stroke:#3D7A3D,color:#fff
67+
classDef optPkg fill:#F0AD4E,stroke:#C79121,color:#fff
68+
classDef storage fill:#9B59B6,stroke:#6C3483,color:#fff
69+
classDef user fill:#E74C3C,stroke:#A93226,color:#fff
70+
71+
class CLI,LAZY metaPkg
72+
class CAPTURE,ML,EVALS,VIEWER corePkg
73+
class GROUNDING,RETRIEVAL,PRIVACY optPkg
74+
class DEMO,MODEL,RESULTS storage
75+
class UI user
76+
```
77+
78+
## Data Flow Pipeline
79+
80+
```mermaid
81+
flowchart LR
82+
subgraph Record["1. Record"]
83+
A[User Demo] --> B[Capture Session]
84+
B --> C[Screenshots + Events]
85+
end
86+
87+
subgraph Store["2. Store"]
88+
C --> D[JSON/Parquet Files]
89+
D --> E[Demo Library]
90+
end
91+
92+
subgraph Train["3. Train"]
93+
E --> F[Data Loading]
94+
F --> G[Model Training]
95+
G --> H[Checkpoint]
96+
end
97+
98+
subgraph Deploy["4. Deploy"]
99+
H --> I[Agent Policy]
100+
I --> J[Inference]
101+
J --> K[Action Replay]
102+
end
103+
104+
subgraph Evaluate["5. Evaluate"]
105+
I --> L[Benchmark Runner]
106+
L --> M[Metrics]
107+
M --> N[Results Report]
108+
end
109+
110+
%% Optional enhancements
111+
GROUND[Grounding] -.-> J
112+
RETRIEVE[Retrieval] -.-> F
113+
PRIV[Privacy] -.-> C
114+
115+
classDef phase fill:#3498DB,stroke:#1A5276,color:#fff
116+
classDef optional fill:#F39C12,stroke:#B7950B,color:#fff
117+
118+
class A,B,C,D,E,F,G,H,I,J,K,L,M,N phase
119+
class GROUND,RETRIEVE,PRIV optional
120+
```
121+
122+
## Package Dependencies
123+
124+
```mermaid
125+
graph TD
126+
OA[openadapt<br/>Meta-package]
127+
128+
OA -->|capture| CAP[openadapt-capture]
129+
OA -->|ml| MLP[openadapt-ml]
130+
OA -->|evals| EVL[openadapt-evals]
131+
OA -->|viewer| VWR[openadapt-viewer]
132+
OA -->|grounding| GRD[openadapt-grounding]
133+
OA -->|retrieval| RET[openadapt-retrieval]
134+
OA -->|privacy| PRV[openadapt-privacy]
135+
136+
%% Core bundle
137+
OA -->|core| CORE[Core Bundle]
138+
CORE --> CAP
139+
CORE --> MLP
140+
CORE --> EVL
141+
CORE --> VWR
142+
143+
%% All bundle
144+
OA -->|all| ALL[Full Bundle]
145+
ALL --> CORE
146+
ALL --> GRD
147+
ALL --> RET
148+
ALL --> PRV
149+
150+
classDef meta fill:#2C3E50,stroke:#1A252F,color:#fff
151+
classDef core fill:#27AE60,stroke:#1E8449,color:#fff
152+
classDef optional fill:#E67E22,stroke:#A04000,color:#fff
153+
classDef bundle fill:#8E44AD,stroke:#5B2C6F,color:#fff
154+
155+
class OA meta
156+
class CAP,MLP,EVL,VWR core
157+
class GRD,RET,PRV optional
158+
class CORE,ALL bundle
159+
```
160+
161+
## Component Details
162+
163+
### Core Packages
164+
165+
| Package | Responsibility | Key Exports |
166+
|---------|---------------|-------------|
167+
| **openadapt-capture** | GUI recording, event capture, storage | `CaptureSession`, `Recorder`, `Action` |
168+
| **openadapt-ml** | Model training, inference, adapters | `QwenVLAdapter`, `Trainer`, `AgentPolicy` |
169+
| **openadapt-evals** | Benchmark evaluation, metrics | `ApiAgent`, `BenchmarkAdapter`, `evaluate_agent_on_benchmark` |
170+
| **openadapt-viewer** | HTML visualization, replay viewer | `PageBuilder`, `HTMLBuilder` |
171+
172+
### Optional Packages
173+
174+
| Package | Responsibility | Use Case |
175+
|---------|---------------|----------|
176+
| **openadapt-grounding** | UI element localization | Improved click accuracy with element detection |
177+
| **openadapt-retrieval** | Multimodal demo search | Find similar demonstrations for few-shot prompting |
178+
| **openadapt-privacy** | PII/PHI scrubbing | Redact sensitive data before storage/training |
179+
180+
## Evaluation Loop
181+
182+
```mermaid
183+
flowchart TB
184+
subgraph Agent["Agent Under Test"]
185+
POLICY[Agent Policy]
186+
API[API Agent<br/>Claude/GPT]
187+
end
188+
189+
subgraph Benchmark["Benchmark System"]
190+
ADAPTER[Benchmark Adapter]
191+
MOCK[Mock Adapter]
192+
LIVE[Live WAA Adapter]
193+
end
194+
195+
subgraph Tasks["Task Execution"]
196+
TASK[Get Task]
197+
OBS[Observe State]
198+
ACT[Execute Action]
199+
CHECK[Check Success]
200+
end
201+
202+
subgraph Metrics["Metrics"]
203+
SUCCESS[Success Rate]
204+
STEPS[Avg Steps]
205+
TIME[Execution Time]
206+
end
207+
208+
POLICY --> ADAPTER
209+
API --> ADAPTER
210+
ADAPTER --> MOCK
211+
ADAPTER --> LIVE
212+
213+
MOCK --> TASK
214+
LIVE --> TASK
215+
TASK --> OBS
216+
OBS --> POLICY
217+
OBS --> API
218+
POLICY --> ACT
219+
API --> ACT
220+
ACT --> CHECK
221+
CHECK -->|next| TASK
222+
CHECK -->|done| SUCCESS
223+
CHECK --> STEPS
224+
CHECK --> TIME
225+
226+
classDef agent fill:#3498DB,stroke:#1A5276,color:#fff
227+
classDef bench fill:#2ECC71,stroke:#1E8449,color:#fff
228+
classDef task fill:#9B59B6,stroke:#6C3483,color:#fff
229+
classDef metric fill:#E74C3C,stroke:#A93226,color:#fff
230+
231+
class POLICY,API agent
232+
class ADAPTER,MOCK,LIVE bench
233+
class TASK,OBS,ACT,CHECK task
234+
class SUCCESS,STEPS,TIME metric
235+
```
236+
237+
## CLI Command Structure
238+
239+
```mermaid
240+
graph LR
241+
OA[openadapt]
242+
243+
OA --> CAP[capture]
244+
OA --> TRN[train]
245+
OA --> EVL[eval]
246+
OA --> SRV[serve]
247+
OA --> VER[version]
248+
OA --> DOC[doctor]
249+
250+
CAP --> CS[start]
251+
CAP --> CT[stop]
252+
CAP --> CL[list]
253+
CAP --> CV[view]
254+
255+
TRN --> TS[start]
256+
TRN --> TST[status]
257+
TRN --> TSP[stop]
258+
259+
EVL --> ER[run]
260+
EVL --> EM[mock]
261+
262+
classDef root fill:#2C3E50,stroke:#1A252F,color:#fff
263+
classDef group fill:#3498DB,stroke:#1A5276,color:#fff
264+
classDef cmd fill:#27AE60,stroke:#1E8449,color:#fff
265+
266+
class OA root
267+
class CAP,TRN,EVL,SRV,VER,DOC group
268+
class CS,CT,CL,CV,TS,TST,TSP,ER,EM cmd
269+
```
270+
271+
## Installation Options
272+
273+
```bash
274+
# Minimal CLI only
275+
pip install openadapt
276+
277+
# Individual packages
278+
pip install openadapt[capture] # GUI capture/recording
279+
pip install openadapt[ml] # ML training and inference
280+
pip install openadapt[evals] # Benchmark evaluation
281+
pip install openadapt[viewer] # HTML visualization
282+
283+
# Optional packages
284+
pip install openadapt[grounding] # UI element localization
285+
pip install openadapt[retrieval] # Demo search/retrieval
286+
pip install openadapt[privacy] # PII/PHI scrubbing
287+
288+
# Bundles
289+
pip install openadapt[core] # capture + ml + evals + viewer
290+
pip install openadapt[all] # Everything
291+
```
292+
293+
---
294+
295+
*This architecture enables independent development and versioning of each component while maintaining a unified CLI experience.*

0 commit comments

Comments
 (0)