|
| 1 | +# OpenAdapt Architecture |
| 2 | + |
| 3 | +OpenAdapt v1.0+ uses a **modular meta-package architecture** where the main `openadapt` package provides a unified CLI and depends on focused sub-packages. |
| 4 | + |
| 5 | +## System Overview |
| 6 | + |
| 7 | +```mermaid |
| 8 | +flowchart TB |
| 9 | + subgraph User["User"] |
| 10 | + UI[Desktop/Web GUI] |
| 11 | + end |
| 12 | +
|
| 13 | + subgraph OpenAdapt["OpenAdapt Meta-Package"] |
| 14 | + CLI[openadapt CLI] |
| 15 | + LAZY[Lazy Imports] |
| 16 | + end |
| 17 | +
|
| 18 | + subgraph Core["Core Packages"] |
| 19 | + CAPTURE[openadapt-capture] |
| 20 | + ML[openadapt-ml] |
| 21 | + EVALS[openadapt-evals] |
| 22 | + VIEWER[openadapt-viewer] |
| 23 | + end |
| 24 | +
|
| 25 | + subgraph Optional["Optional Packages"] |
| 26 | + GROUNDING[openadapt-grounding] |
| 27 | + RETRIEVAL[openadapt-retrieval] |
| 28 | + PRIVACY[openadapt-privacy] |
| 29 | + end |
| 30 | +
|
| 31 | + subgraph Storage["Storage"] |
| 32 | + DEMO[(Demonstration<br/>JSON/Parquet)] |
| 33 | + MODEL[(Model<br/>Checkpoints)] |
| 34 | + RESULTS[(Evaluation<br/>Results)] |
| 35 | + end |
| 36 | +
|
| 37 | + %% User interactions |
| 38 | + UI --> CAPTURE |
| 39 | +
|
| 40 | + %% CLI orchestration |
| 41 | + CLI --> CAPTURE |
| 42 | + CLI --> ML |
| 43 | + CLI --> EVALS |
| 44 | + CLI --> VIEWER |
| 45 | +
|
| 46 | + %% Lazy loading |
| 47 | + LAZY -.-> GROUNDING |
| 48 | + LAZY -.-> RETRIEVAL |
| 49 | + LAZY -.-> PRIVACY |
| 50 | +
|
| 51 | + %% Data flow |
| 52 | + CAPTURE --> DEMO |
| 53 | + DEMO --> ML |
| 54 | + ML --> MODEL |
| 55 | + MODEL --> EVALS |
| 56 | + EVALS --> RESULTS |
| 57 | + DEMO --> VIEWER |
| 58 | +
|
| 59 | + %% Optional integrations |
| 60 | + GROUNDING -.-> ML |
| 61 | + RETRIEVAL -.-> ML |
| 62 | + PRIVACY -.-> CAPTURE |
| 63 | + PRIVACY -.-> VIEWER |
| 64 | +
|
| 65 | + classDef metaPkg fill:#4A90D9,stroke:#2E5A8B,color:#fff |
| 66 | + classDef corePkg fill:#5CB85C,stroke:#3D7A3D,color:#fff |
| 67 | + classDef optPkg fill:#F0AD4E,stroke:#C79121,color:#fff |
| 68 | + classDef storage fill:#9B59B6,stroke:#6C3483,color:#fff |
| 69 | + classDef user fill:#E74C3C,stroke:#A93226,color:#fff |
| 70 | +
|
| 71 | + class CLI,LAZY metaPkg |
| 72 | + class CAPTURE,ML,EVALS,VIEWER corePkg |
| 73 | + class GROUNDING,RETRIEVAL,PRIVACY optPkg |
| 74 | + class DEMO,MODEL,RESULTS storage |
| 75 | + class UI user |
| 76 | +``` |
| 77 | + |
| 78 | +## Data Flow Pipeline |
| 79 | + |
| 80 | +```mermaid |
| 81 | +flowchart LR |
| 82 | + subgraph Record["1. Record"] |
| 83 | + A[User Demo] --> B[Capture Session] |
| 84 | + B --> C[Screenshots + Events] |
| 85 | + end |
| 86 | +
|
| 87 | + subgraph Store["2. Store"] |
| 88 | + C --> D[JSON/Parquet Files] |
| 89 | + D --> E[Demo Library] |
| 90 | + end |
| 91 | +
|
| 92 | + subgraph Train["3. Train"] |
| 93 | + E --> F[Data Loading] |
| 94 | + F --> G[Model Training] |
| 95 | + G --> H[Checkpoint] |
| 96 | + end |
| 97 | +
|
| 98 | + subgraph Deploy["4. Deploy"] |
| 99 | + H --> I[Agent Policy] |
| 100 | + I --> J[Inference] |
| 101 | + J --> K[Action Replay] |
| 102 | + end |
| 103 | +
|
| 104 | + subgraph Evaluate["5. Evaluate"] |
| 105 | + I --> L[Benchmark Runner] |
| 106 | + L --> M[Metrics] |
| 107 | + M --> N[Results Report] |
| 108 | + end |
| 109 | +
|
| 110 | + %% Optional enhancements |
| 111 | + GROUND[Grounding] -.-> J |
| 112 | + RETRIEVE[Retrieval] -.-> F |
| 113 | + PRIV[Privacy] -.-> C |
| 114 | +
|
| 115 | + classDef phase fill:#3498DB,stroke:#1A5276,color:#fff |
| 116 | + classDef optional fill:#F39C12,stroke:#B7950B,color:#fff |
| 117 | +
|
| 118 | + class A,B,C,D,E,F,G,H,I,J,K,L,M,N phase |
| 119 | + class GROUND,RETRIEVE,PRIV optional |
| 120 | +``` |
| 121 | + |
| 122 | +## Package Dependencies |
| 123 | + |
| 124 | +```mermaid |
| 125 | +graph TD |
| 126 | + OA[openadapt<br/>Meta-package] |
| 127 | +
|
| 128 | + OA -->|capture| CAP[openadapt-capture] |
| 129 | + OA -->|ml| MLP[openadapt-ml] |
| 130 | + OA -->|evals| EVL[openadapt-evals] |
| 131 | + OA -->|viewer| VWR[openadapt-viewer] |
| 132 | + OA -->|grounding| GRD[openadapt-grounding] |
| 133 | + OA -->|retrieval| RET[openadapt-retrieval] |
| 134 | + OA -->|privacy| PRV[openadapt-privacy] |
| 135 | +
|
| 136 | + %% Core bundle |
| 137 | + OA -->|core| CORE[Core Bundle] |
| 138 | + CORE --> CAP |
| 139 | + CORE --> MLP |
| 140 | + CORE --> EVL |
| 141 | + CORE --> VWR |
| 142 | +
|
| 143 | + %% All bundle |
| 144 | + OA -->|all| ALL[Full Bundle] |
| 145 | + ALL --> CORE |
| 146 | + ALL --> GRD |
| 147 | + ALL --> RET |
| 148 | + ALL --> PRV |
| 149 | +
|
| 150 | + classDef meta fill:#2C3E50,stroke:#1A252F,color:#fff |
| 151 | + classDef core fill:#27AE60,stroke:#1E8449,color:#fff |
| 152 | + classDef optional fill:#E67E22,stroke:#A04000,color:#fff |
| 153 | + classDef bundle fill:#8E44AD,stroke:#5B2C6F,color:#fff |
| 154 | +
|
| 155 | + class OA meta |
| 156 | + class CAP,MLP,EVL,VWR core |
| 157 | + class GRD,RET,PRV optional |
| 158 | + class CORE,ALL bundle |
| 159 | +``` |
| 160 | + |
| 161 | +## Component Details |
| 162 | + |
| 163 | +### Core Packages |
| 164 | + |
| 165 | +| Package | Responsibility | Key Exports | |
| 166 | +|---------|---------------|-------------| |
| 167 | +| **openadapt-capture** | GUI recording, event capture, storage | `CaptureSession`, `Recorder`, `Action` | |
| 168 | +| **openadapt-ml** | Model training, inference, adapters | `QwenVLAdapter`, `Trainer`, `AgentPolicy` | |
| 169 | +| **openadapt-evals** | Benchmark evaluation, metrics | `ApiAgent`, `BenchmarkAdapter`, `evaluate_agent_on_benchmark` | |
| 170 | +| **openadapt-viewer** | HTML visualization, replay viewer | `PageBuilder`, `HTMLBuilder` | |
| 171 | + |
| 172 | +### Optional Packages |
| 173 | + |
| 174 | +| Package | Responsibility | Use Case | |
| 175 | +|---------|---------------|----------| |
| 176 | +| **openadapt-grounding** | UI element localization | Improved click accuracy with element detection | |
| 177 | +| **openadapt-retrieval** | Multimodal demo search | Find similar demonstrations for few-shot prompting | |
| 178 | +| **openadapt-privacy** | PII/PHI scrubbing | Redact sensitive data before storage/training | |
| 179 | + |
| 180 | +## Evaluation Loop |
| 181 | + |
| 182 | +```mermaid |
| 183 | +flowchart TB |
| 184 | + subgraph Agent["Agent Under Test"] |
| 185 | + POLICY[Agent Policy] |
| 186 | + API[API Agent<br/>Claude/GPT] |
| 187 | + end |
| 188 | +
|
| 189 | + subgraph Benchmark["Benchmark System"] |
| 190 | + ADAPTER[Benchmark Adapter] |
| 191 | + MOCK[Mock Adapter] |
| 192 | + LIVE[Live WAA Adapter] |
| 193 | + end |
| 194 | +
|
| 195 | + subgraph Tasks["Task Execution"] |
| 196 | + TASK[Get Task] |
| 197 | + OBS[Observe State] |
| 198 | + ACT[Execute Action] |
| 199 | + CHECK[Check Success] |
| 200 | + end |
| 201 | +
|
| 202 | + subgraph Metrics["Metrics"] |
| 203 | + SUCCESS[Success Rate] |
| 204 | + STEPS[Avg Steps] |
| 205 | + TIME[Execution Time] |
| 206 | + end |
| 207 | +
|
| 208 | + POLICY --> ADAPTER |
| 209 | + API --> ADAPTER |
| 210 | + ADAPTER --> MOCK |
| 211 | + ADAPTER --> LIVE |
| 212 | +
|
| 213 | + MOCK --> TASK |
| 214 | + LIVE --> TASK |
| 215 | + TASK --> OBS |
| 216 | + OBS --> POLICY |
| 217 | + OBS --> API |
| 218 | + POLICY --> ACT |
| 219 | + API --> ACT |
| 220 | + ACT --> CHECK |
| 221 | + CHECK -->|next| TASK |
| 222 | + CHECK -->|done| SUCCESS |
| 223 | + CHECK --> STEPS |
| 224 | + CHECK --> TIME |
| 225 | +
|
| 226 | + classDef agent fill:#3498DB,stroke:#1A5276,color:#fff |
| 227 | + classDef bench fill:#2ECC71,stroke:#1E8449,color:#fff |
| 228 | + classDef task fill:#9B59B6,stroke:#6C3483,color:#fff |
| 229 | + classDef metric fill:#E74C3C,stroke:#A93226,color:#fff |
| 230 | +
|
| 231 | + class POLICY,API agent |
| 232 | + class ADAPTER,MOCK,LIVE bench |
| 233 | + class TASK,OBS,ACT,CHECK task |
| 234 | + class SUCCESS,STEPS,TIME metric |
| 235 | +``` |
| 236 | + |
| 237 | +## CLI Command Structure |
| 238 | + |
| 239 | +```mermaid |
| 240 | +graph LR |
| 241 | + OA[openadapt] |
| 242 | +
|
| 243 | + OA --> CAP[capture] |
| 244 | + OA --> TRN[train] |
| 245 | + OA --> EVL[eval] |
| 246 | + OA --> SRV[serve] |
| 247 | + OA --> VER[version] |
| 248 | + OA --> DOC[doctor] |
| 249 | +
|
| 250 | + CAP --> CS[start] |
| 251 | + CAP --> CT[stop] |
| 252 | + CAP --> CL[list] |
| 253 | + CAP --> CV[view] |
| 254 | +
|
| 255 | + TRN --> TS[start] |
| 256 | + TRN --> TST[status] |
| 257 | + TRN --> TSP[stop] |
| 258 | +
|
| 259 | + EVL --> ER[run] |
| 260 | + EVL --> EM[mock] |
| 261 | +
|
| 262 | + classDef root fill:#2C3E50,stroke:#1A252F,color:#fff |
| 263 | + classDef group fill:#3498DB,stroke:#1A5276,color:#fff |
| 264 | + classDef cmd fill:#27AE60,stroke:#1E8449,color:#fff |
| 265 | +
|
| 266 | + class OA root |
| 267 | + class CAP,TRN,EVL,SRV,VER,DOC group |
| 268 | + class CS,CT,CL,CV,TS,TST,TSP,ER,EM cmd |
| 269 | +``` |
| 270 | + |
| 271 | +## Installation Options |
| 272 | + |
| 273 | +```bash |
| 274 | +# Minimal CLI only |
| 275 | +pip install openadapt |
| 276 | + |
| 277 | +# Individual packages |
| 278 | +pip install openadapt[capture] # GUI capture/recording |
| 279 | +pip install openadapt[ml] # ML training and inference |
| 280 | +pip install openadapt[evals] # Benchmark evaluation |
| 281 | +pip install openadapt[viewer] # HTML visualization |
| 282 | + |
| 283 | +# Optional packages |
| 284 | +pip install openadapt[grounding] # UI element localization |
| 285 | +pip install openadapt[retrieval] # Demo search/retrieval |
| 286 | +pip install openadapt[privacy] # PII/PHI scrubbing |
| 287 | + |
| 288 | +# Bundles |
| 289 | +pip install openadapt[core] # capture + ml + evals + viewer |
| 290 | +pip install openadapt[all] # Everything |
| 291 | +``` |
| 292 | + |
| 293 | +--- |
| 294 | + |
| 295 | +*This architecture enables independent development and versioning of each component while maintaining a unified CLI experience.* |
0 commit comments