feat: Phase 2E — Vibe app action wiring (change_background proof-of-concept)#39
Closed
stozo04 wants to merge 2 commits intoMiniMax-AI:mainfrom
Closed
feat: Phase 2E — Vibe app action wiring (change_background proof-of-concept)#39stozo04 wants to merge 2 commits intoMiniMax-AI:mainfrom
stozo04 wants to merge 2 commits intoMiniMax-AI:mainfrom
Conversation
…TS (#1) * feat(vibe): scaffold BookWriter and MemoryVault stubs Two new Vibe apps for Kayley's cockpit v1: - BookWriter (appId 15) — Steven's novel workspace - MemoryVault (appId 16) — captured moments, reflections, promises Each follows the EvidenceVault pattern: index.tsx + scss + {name}_en/cn meta.yaml + guide.md. Registered in routers/index.tsx and lib/appRegistry.ts so they appear on the desktop and in list_apps. UI is a placeholder ("coming soon") — real data wiring comes later. Existing Twitter/Album/Diary/Email/MusicPlayer kept as-is. * feat(album): wire Album to 542 Kayley selfies via media symlink - Junction public/kayley-selfies -> Kayley_Cowork/media/selfies (542 PNG/JPG) - Generated public/kayley-selfies-index.json (parses timestamp + location from filenames, sorts newest-first) - Album now fetches the static index at runtime and merges with any cloud-FS agent-added images (dedup by id) - Update meta.yaml so the character understands the album is Kayley's selfie collection (outfits, locations, moods) instead of a generic gallery - .gitignore the junction and generated index so 542 binary files stay in Kayley_Cowork, not OpenRoom * feat(memoryvault): wire to captured_moments, weekly evaluations, and promises * feat(vibe): install /vibe command + enhance MusicPlayer (Songs From Kayley) + BookWriter v1 /vibe slash command installed in Kayley-Cowork at .claude/commands/vibe.md (patched to resolve paths against OPENROOM_ROOT so it can be invoked from a Kayley-Cowork Claude Code session and still write into the correct OpenRoom tree). Workflow stage/rule files are sourced directly from OpenRoom to prevent drift. MusicPlayer now ships a featured "Songs From Kayley" playlist with Landslide (Fleetwood Mac) pinned at index 0 and two companion tracks (The Night We Met, Harvest Moon). Real Amazon Music API integration is tracked as a follow-on project; this is the mocked seed for now. BookWriter replaced its "coming soon" placeholder with a chapter-list UI that loads from /kayley-book/index.json. When no manifest exists it shows a graceful empty state with instructions ("drop markdown in Kayley_Cowork/book/chapters/ to start") plus a preview of the list UI. No book chapters exist on disk yet, so no junction was created. Also included: orphaned Diary wiring from iter 5 (journal + captured moments) that had not been committed. * feat(email): wire Email panel to gog Gmail CLI via static snapshot Iter 7 of the Kayley Vibe app wiring. Email app now reads Steven's real Gmail inbox (read-only) instead of showing an empty state. - scripts/refresh-email-index.mjs: calls `gog gmail search in:inbox` + `gog gmail get <id>` to build a static snapshot at apps/webuiapps/public/kayley-email-index.json (gitignored, PII). - Email/index.tsx: on init, fetches the snapshot and merges entries not already in the local FS (agent-authored emails still take precedence by id). Graceful empty state if the snapshot is missing. - email_en/meta.yaml: updated description so the agent understands this is read-only for v1; sending happens via the gog CLI in Kayley-Cowork. - .gitignore: added kayley-email-index.json (contains PII). Verified locally via Chrome: 40 messages loaded, 38 unread badge, real senders rendering (LinkedIn, BCBS Texas, Fidelity, DFW Parking, SelfStorageAuction). Phase 1 status: 6/7 apps wired. Twitter remaining. * feat(twitter): wire Twitter as in-world Kayley feed (Phase 1 finale) Seed Twitter from Kayley's real artifacts so the feed reads like she's been posting throughout the day: - scripts/build-twitter-feed.mjs: transforms captured moments (milestone posts with "line that stays"), private journal entries (thinking-out- loud snippets with hashtags inferred from emotion headers), and every 8th selfie from kayley-selfies-index.json (scene / outfit posts with the selfie image attached). Templated friend replies from Jessica Park, Chloe Parker, Emmy Carter, and Mateo Rivera are sprinkled onto ~30% of posts using a deterministic seeded RNG so re-runs are stable. - apps/webuiapps/src/pages/Twitter/index.tsx: added optional Post.image field, renders inline via new .postImage style (rounded card, 480px max-height). On first mount (when the FS has no posts yet), fetches /kayley-twitter-feed.json and seeds state so agents and Steven land on a populated feed instead of the empty state. - twitter_en/meta.yaml: description rewritten so the LLM knows this is Kayley's social feed — her posts + friend replies — not a generic social app. Agent can still CREATE_POST / LIKE_POST / COMMENT_POST normally. Current run: 14 moment posts, 24 journal posts, 68 selfie posts (105 total), 56 friend replies. The feed JSON is gitignored (personal content — regenerate with `node scripts/build-twitter-feed.mjs`). * fix(build): wrap NODE_ENV= scripts in cross-env for Windows compatibility * feat(2B-2D): wire ChatPanel to Kayley brain via websocket channel - Add useKayleyChannel hook: connects to ws://localhost:5180 (same channel as dashboard), handles text send, STT voice input via mic + ScriptProcessorNode at 16kHz, TTS audio playback via Web Audio API at 24kHz, and STT draft review before sending. - ChatPanel: routes messages to Kayley brain when connected (bypasses local LLM). Falls back to local LLM if server is unreachable. Adds mic button (voice mode), STT draft confirmation bar, and a green dot connection indicator in the header. - SCSS: adds .micBtn/.micActive pulse, .sttDraftBar, .kayleyDot. Lifted audio/mic code directly from dashboard/src/main.js — same protocol, same PCM encoding, same Web Audio scheduling. Dashboard continues to work independently. * fix(kayley-channel): address PR #1 review — AudioContext gesture init, send timeout, race + cleanup, no silent catches, env override - Prime TTS AudioContext from sendText() user gesture (Chrome/Safari autoplay) - Add 90s send-timeout in ChatPanel so stalled Kayley brain unlocks UI - Clear reconnectTimerRef before every reschedule (catch + onclose) - Expand unmount cleanup: close audio + mic AudioContexts, stop mic tracks, clear buffer timer (prevents LED stuck on / AudioContext cap exhaustion) - Replace silent catches: ws.onerror logs warning, JSON.parse logs truncated raw - Log mic AudioContext actual sampleRate (Firefox doesn't honor request) - KAYLEY_WS_URL now env-overridable via VITE_KAYLEY_WS_URL - STT confirm button routes through kayley.confirmDraft() for consistent side-effects
…Phase 2E)
Previously the Kayley WebSocket channel was text/voice/image only. This wires
OpenRoom's existing app_action dispatch layer into the Kayley brain so she
can drive the desktop the same way Aoi can natively. v1 scope:
change_background (os/SET_WALLPAPER) proof-of-concept.
Flow: Kayley calls the vibe_action MCP tool on the websocket-channel server
-> server broadcasts {type:'vibe_action', id, app_name, action_type, params}
-> useKayleyChannel receives, runs it through resolveAppAction +
dispatchAgentAction (same path the local LLM uses), echoes
{type:'vibe_action_result', id, result} back to the server.
- useKayleyChannel.ts: new dispatchVibeAction helper + ws.onmessage branch.
seedMetaFiles + loadActionsFromMeta are called on first dispatch to stay
forward-compatible with per-app actions; OS actions (SET_WALLPAPER) work
without them today.
- docs/PHASE-2E-RECON.md: full recon of Aoi's action protocol -- catalog
discovery, tool format, dispatch, sync/async behavior, round-trip state.
(Force-added; docs/ is normally gitignored but these are deliverables.)
- docs/PHASE-2E-DESIGN.md: bridge design, failure modes, why we inject the
catalog server-side via MCP tool schema rather than streaming it.
- docs/PHASE-2E-TESTING.md: manual test plan for the end-to-end wallpaper
change, plus failure-mode matrix.
No behavior change for existing Kayley text/voice/image paths. ChatPanel is
untouched. Paired with Kayley-Cowork kayley-v2e branch (vibe_action MCP
tool). Refs Phase 2 issue #78.
Author
|
Retargeting to stozo04/OpenRoom (the actual project repo). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires Kayley's brain (Claude Opus over the websocket-channel MCP server) into OpenRoom's existing
app_actiondispatch pipeline so she can drive the Vibe desktop the same way Aoi can natively. v1 scope:change_background(os/SET_WALLPAPER) end-to-end.useKayleyChannel.ts— handles{type:'vibe_action'}inbound messages, runs them throughresolveAppAction+dispatchAgentAction(same path the in-app LLM uses), echoes the result back as{type:'vibe_action_result'}.docs/PHASE-2E-RECON.md— full recon of Aoi's action protocol (meta.yaml schema, discovery flow, tool format, dispatch layer, sync/async behavior, round-trip state).docs/PHASE-2E-DESIGN.md— bridge design, rationale for server-side catalog injection, failure modes.docs/PHASE-2E-TESTING.md— manual test plan + failure-mode matrix.Paired with Kayley-Cowork PR on branch
kayley-v2e: adds thevibe_actionMCP tool + pending-action bookkeeping inplugins/websocket-channel/server.ts.No behavior change for existing Kayley text/voice/image flows. ChatPanel is untouched.
Refs Phase 2 issue #78. Co-designed after Steven confirmed Aoi's
change_backgroundflow works 2026-04-15.Test plan
vibe_actionis exposed alongsidereplyandcheck_inbox.docs/PHASE-2E-TESTING.md: no browser client connected, malformed params, unknown app_name, dispatch timeout.Hard scope
os/SET_WALLPAPERis wired for v1. Per-app actions (Album REFRESH, etc.) are out of scope and deferred to a follow-up PR.