Skip to content

feat: Phase 2E — Vibe app action wiring (change_background proof-of-concept) #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
stozo04 wants to merge 2 commits into from
Closed

feat: Phase 2E — Vibe app action wiring (change_background proof-of-concept) #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3096eb0
feat(phase-2B-2D): wire ChatPanel to Kayley brain — WebSocket, STT, T…
stozo04 Apr 15, 2026
fdceaa5
feat(kayley-channel): bridge vibe_action dispatch from Kayley brain (…
stozo04 Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,24 @@ CLAUDE.local.md
# Cursor
.cursor/

# Kayley data (symlinked/generated — lives in Kayley_Cowork + ~/.kayley-journal)
apps/webuiapps/public/kayley-selfies
apps/webuiapps/public/kayley-selfies/
apps/webuiapps/public/kayley-selfies-index.json
apps/webuiapps/public/kayley-moments
apps/webuiapps/public/kayley-moments/
apps/webuiapps/public/kayley-moments-index.json
apps/webuiapps/public/kayley-journal
apps/webuiapps/public/kayley-journal/
apps/webuiapps/public/kayley-journal-index.json
apps/webuiapps/public/kayley-story
apps/webuiapps/public/kayley-story/
apps/webuiapps/public/kayley-weeks-index.json
apps/webuiapps/public/kayley-promises-index.json
apps/webuiapps/public/kayley-storylines-index.json
apps/webuiapps/public/kayley-email-index.json
apps/webuiapps/public/kayley-twitter-feed.json

# Internal CI/CD & Scripts
scripts/
.gitlab-ci.yml
Expand Down
7 changes: 4 additions & 3 deletions apps/webuiapps/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
"scripts": {
"dev": "vite",
"debug": "vite --debug",
"build:test": "NODE_ENV=test vite build",
"build": "NODE_ENV=production vite build",
"build:analyze": "NODE_ENV=production ANALYZE=analyze vite build",
"build:test": "cross-env NODE_ENV=test vite build",
"build": "cross-env NODE_ENV=production vite build",
"build:analyze": "cross-env NODE_ENV=production ANALYZE=analyze vite build",
"preview": "vite preview",
"test": "vitest run",
"test:watch": "vitest",
Expand All @@ -26,6 +26,7 @@
"@anthropic-ai/claude-agent-sdk": "^0.2.72",
"@vitest/coverage-istanbul": "^1.6.1",
"@vitest/coverage-v8": "^1.6.1",
"cross-env": "^10.1.0",
"happy-dom": "^14.0.0",
"jsdom": "^25.0.0",
"vitest": "^1.6.1"
Expand Down
90 changes: 90 additions & 0 deletions apps/webuiapps/src/components/ChatPanel/index.module.scss
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,3 +452,93 @@
color: rgba(250, 234, 95, 0.7);
margin-top: 4px;
}

// ── Kayley channel additions ────────────────────────────────────────────────

.kayleyDot {
display: inline-block;
width: 6px;
height: 6px;
border-radius: 50%;
background: #4ade80;
margin-left: 6px;
vertical-align: middle;
}

.sttDraftBar {
padding: 8px 16px;
border-top: 1px solid rgba(255, 255, 255, 0.06);
background: rgba(250, 234, 95, 0.06);
display: flex;
align-items: center;
gap: 8px;
}

.sttDraftText {
flex: 1;
font-size: 13px;
color: rgba(255, 255, 255, 0.8);
font-style: italic;
}

.sttDraftConfirm {
background: #faea5f;
color: #121214;
border: none;
border-radius: 6px;
padding: 4px 10px;
cursor: pointer;
font-weight: 600;
font-size: 12px;
white-space: nowrap;

&:hover {
background: #f5e04a;
}
}

.sttDraftDismiss {
background: transparent;
color: rgba(255, 255, 255, 0.5);
border: 1px solid rgba(255, 255, 255, 0.15);
border-radius: 6px;
padding: 4px 8px;
cursor: pointer;
font-size: 12px;

&:hover {
color: rgba(255, 255, 255, 0.8);
border-color: rgba(255, 255, 255, 0.3);
}
}

.micBtn {
background: transparent;
color: rgba(255, 255, 255, 0.6);
border: 1px solid rgba(255, 255, 255, 0.15);
border-radius: 8px;
padding: 8px 10px;
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
transition: all 0.15s;
flex-shrink: 0;

&:hover {
color: rgba(255, 255, 255, 0.9);
border-color: rgba(255, 255, 255, 0.3);
}
}

.micActive {
background: rgba(239, 68, 68, 0.15);
color: #ef4444;
border-color: rgba(239, 68, 68, 0.4);
animation: micPulse 1s ease-in-out infinite;
}

@keyframes micPulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.6; }
}
136 changes: 133 additions & 3 deletions apps/webuiapps/src/components/ChatPanel/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ import {
ChevronRight,
Pencil,
List,
Mic,
} from 'lucide-react';
import { chat, loadConfig, loadConfigSync, saveConfig, type ChatMessage } from '@/lib/llmClient';
import { useKayleyChannel } from '@/hooks/useKayleyChannel';
import {
PROVIDER_MODELS,
getDefaultProviderConfig,
Expand Down Expand Up @@ -85,6 +87,10 @@ import CharacterPanel from './CharacterPanel';
import ModPanel from './ModPanel';
import styles from './index.module.scss';

// If the Kayley brain (MCP) doesn't respond within this window we release the
// UI so `loading` can't stick forever.
const KAYLEY_SEND_TIMEOUT_MS = 90_000;

// ---------------------------------------------------------------------------
// Extended DisplayMessage with character-specific fields
// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -459,6 +465,14 @@ const ChatPanel: React.FC<{
const suggestedRepliesRef = useRef(suggestedReplies);
suggestedRepliesRef.current = suggestedReplies;

// ── Kayley channel (ws://localhost:5180) ──────────────────────
// When connected, bypasses local LLM and routes all chat through the
// Claude Opus brain with full MCPs + memory. Falls back to local LLM if
// Kayley server is not running.
const kayley = useKayleyChannel();
const kayleyRef = useRef(kayley);
kayleyRef.current = kayley;

// Debounced save
const saveTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);

Expand Down Expand Up @@ -631,6 +645,37 @@ const ChatPanel: React.FC<{
setMessages((prev) => [...prev, msg]);
}, []);

// ── Kayley send timeout ───────────────────────────────────────
// If the Kayley brain stalls (MCP doesn't reply), clear `loading` after
// KAYLEY_SEND_TIMEOUT_MS so the UI doesn't lock forever. Cleared whenever a
// new reply arrives or the component unmounts.
const kayleySendTimeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);

const clearKayleySendTimeout = useCallback(() => {
if (kayleySendTimeoutRef.current) {
clearTimeout(kayleySendTimeoutRef.current);
kayleySendTimeoutRef.current = null;
}
}, []);

useEffect(() => {
return () => clearKayleySendTimeout();
}, [clearKayleySendTimeout]);

// ── Kayley response handler ───────────────────────────────────
// Each new latestMessage is a unique object (different timestamp), so this
// effect fires exactly once per incoming reply.
useEffect(() => {
if (!kayley.latestMessage) return;
clearKayleySendTimeout();
addMessage({
id: String(kayley.latestMessage.timestamp),
role: 'assistant',
content: kayley.latestMessage.text,
});
setLoading(false);
}, [kayley.latestMessage, addMessage, clearKayleySendTimeout]);

const configRef = useRef(config);
configRef.current = config;
const imageGenConfigRef = useRef(imageGenConfig);
Expand Down Expand Up @@ -703,11 +748,39 @@ const ChatPanel: React.FC<{
return unsubscribe;
}, [processActionQueue]);

// Send message
// Send message — routes to Kayley brain if connected, local LLM otherwise
const handleSend = useCallback(
async (overrideText?: string) => {
const text = overrideText ?? input.trim();
if (!text || loading) return;

const kRef = kayleyRef.current;

if (kRef.connected) {
// ── Kayley brain mode — bypass local LLM ────────────────
if (!overrideText) setInput('');
setSuggestedReplies([]);
addMessage({ id: String(Date.now()), role: 'user', content: text });
setLoading(true);
kRef.sendText(text);

// 90-second stall guard — if MCP doesn't reply, release the UI and
// surface an inline notice. Cleared when latestMessage arrives or on unmount.
clearKayleySendTimeout();
kayleySendTimeoutRef.current = setTimeout(() => {
kayleySendTimeoutRef.current = null;
setLoading(false);
addMessage({
id: String(Date.now()),
role: 'assistant',
content:
"Kayley's thinking took too long — try again or check the connection.",
});
}, KAYLEY_SEND_TIMEOUT_MS);
return;
}

// ── Local LLM mode (fallback when Kayley is not running) ──
if (!hasUsableLLMConfig(config)) {
setShowSettings(true);
return;
Expand Down Expand Up @@ -740,7 +813,7 @@ const ChatPanel: React.FC<{
setLoading(false);
}
},
[input, loading, config, chatHistory, addMessage],
[input, loading, config, chatHistory, addMessage, clearKayleySendTimeout],
);

// Core conversation loop
Expand Down Expand Up @@ -1064,6 +1137,9 @@ const ChatPanel: React.FC<{
style={{ cursor: 'pointer' }}
>
<span className={styles.characterName}>{character.character_name}</span>
{kayley.connected && (
<span className={styles.kayleyDot} title="Connected to Kayley brain" />
)}
<ChevronRight size={14} style={{ color: 'rgba(255,255,255,0.4)' }} />
</div>
<div className={styles.headerActions}>
Expand Down Expand Up @@ -1106,7 +1182,9 @@ const ChatPanel: React.FC<{
<div className={styles.messages} data-testid="chat-messages">
{messages.length === 0 && (
<div className={styles.emptyState}>
{hasUsableLLMConfig(config)
{kayley.connected
? 'Kayley is ready — type anything or tap the mic'
: hasUsableLLMConfig(config)
? `${character.character_name} is ready to chat...`
: 'Click the gear icon to configure your LLM connection'}
</div>
Expand Down Expand Up @@ -1148,6 +1226,48 @@ const ChatPanel: React.FC<{
</div>
)}

{/* STT draft confirmation bar — shown after Whisper transcribes mic audio */}
{kayley.sttDraft !== null && (
<div className={styles.sttDraftBar}>
<span className={styles.sttDraftText}>{kayley.sttDraft}</span>
<button
className={styles.sttDraftConfirm}
onClick={() => {
// Confirm via the hook so its `confirmDraft` side-effects
// (resetPlayback + sendText + draft clear) stay co-located.
// We still do UI bookkeeping here so the user bubble appears
// and the send-stall guard is armed.
const draft = kayley.sttDraft;
if (!draft || loading) return;
addMessage({ id: String(Date.now()), role: 'user', content: draft });
setLoading(true);
clearKayleySendTimeout();
kayleySendTimeoutRef.current = setTimeout(() => {
kayleySendTimeoutRef.current = null;
setLoading(false);
addMessage({
id: String(Date.now()),
role: 'assistant',
content:
"Kayley's thinking took too long — try again or check the connection.",
});
}, KAYLEY_SEND_TIMEOUT_MS);
kayley.confirmDraft();
}}
data-testid="stt-confirm-btn"
>
Send
</button>
<button
className={styles.sttDraftDismiss}
onClick={kayley.dismissDraft}
data-testid="stt-dismiss-btn"
>
✕
</button>
</div>
)}

<div className={styles.inputArea}>
<textarea
className={styles.input}
Expand All @@ -1167,6 +1287,16 @@ const ChatPanel: React.FC<{
>
Send
</button>
{kayley.connected && (
<button
className={`${styles.micBtn} ${kayley.isRecording ? styles.micActive : ''}`}
onClick={() => kayley.isRecording ? kayley.stopVoice() : kayley.startVoice()}
title={kayley.isRecording ? 'Stop recording' : 'Voice input (Kayley STT)'}
data-testid="mic-btn"
>
<Mic size={16} />
</button>
)}
</div>
</div>
</div>
Expand Down
Loading