👁 EyeTrack

Real-time gaze tracking with affordable IR cameras. Runs in the browser, supports dual-eye stereo tracking, and works on macOS, Windows, and Linux.

Built on OpenCV pupil detection and polynomial gaze mapping, wrapped in a modern web UI with live video, 3D eye visualization, heatmaps, and gaze trails.

eyetrack-demo-10mb.mp4

---

🛒 Hardware

All you need to get started:

• IR camera — GC0308 USB infrared camera module (1 for single-eye, 2 for dual-eye)
• IR LED — optional, improves pupil contrast in low light
• Glasses frame — any frame to mount cameras near the eyes
• Hot glue gun + glue sticks — to secure cameras onto the frame
• USB cable — to connect cameras to your computer

That's it. No special boards, no soldering required.

 

---

✨ Features

• Dual-eye tracking — up to 2 IR cameras, each assigned to left/right eye
• Three tracking modes — Classic (Orlosky algorithm), Enhanced (EWMA), Screen (direct mapping)
• 9-point gaze calibration — Vision Pro-inspired, with stability checking and audio feedback
• Bounds calibration — learns valid pupil range to filter blinks and outliers
• Live dashboard — annotated video feeds, real-time metrics, pupil size sparkline
• 3D eye model — dual wireframe eyeballs with gaze direction, mirror/anatomical view toggle
• Gaze cursor — fused screen position from both eyes, weighted by confidence and calibration accuracy
• Heatmap & trail — fullscreen gaze visualizations with theme-aware rendering
• Camera binding — identifies cameras by hardware ID, survives reboots
• Dark / light theme — system-aware with manual toggle
• Latency metrics — real-time processing, network, and frame interval monitoring in the header
• Hosted frontend — eyetrack.thebinwang.com, no build step needed
• Persistent config — all settings, calibrations, and camera assignments saved to disk

---

🏗 Architecture

graph TB
    subgraph Hardware
        CAM1[IR Camera L]
        CAM2[IR Camera R]
    end

    subgraph Backend["Backend — FastAPI + OpenCV"]
        CM[CameraManager<br/>Background capture threads]
        FP[FrameProcessor<br/>Pupil detection pipeline]
        BL[Broadcast Loop<br/>WebSocket streaming]
        TR[TrackerRegistry<br/>Multi-camera state]
        PS[Persistence<br/>JSON config file]
        API[REST API<br/>Settings, calibration, cameras]
    end

    subgraph Frontend["Frontend — React + Three.js"]
        WS[WebSocket Client]
        APP[App State<br/>Trackers, calibrations, fused gaze]
        VF[VideoFeed + Metrics<br/>Per-eye cards]
        EM[EyeModel3D<br/>Dual wireframe eyeballs]
        CW[CalibrationWizard<br/>9-point + bounds]
        VIZ[Heatmap / Trail<br/>Canvas visualizations]
        GC[GazeCursor<br/>Fused gaze overlay]
    end

    CAM1 --> CM
    CAM2 --> CM
    CM --> FP
    FP --> BL
    TR --> FP
    BL -->|WebSocket frames + tracking| WS
    API -->|REST| APP
    PS <--> TR
    WS --> APP
    APP --> VF
    APP --> EM
    APP --> CW
    APP --> VIZ
    APP --> GC

Loading

---

🚀 Quick Start

Install uv, then:

uv run eyetrack

Open eyetrack.thebinwang.com — the frontend connects to your local backend automatically.

Self-host the frontend for offline use:

cd web/frontend && pnpm install && pnpm build && cd ../..
uv run eyetrack    # Open http://localhost:8100

Development

cd web/frontend
pnpm install
pnpm dev          # Dev server at http://localhost:5173

---

📂 Project Structure

├── src/                        # Core algorithms
│   └── pupil_detector.py       # Cascaded thresholding + ellipse fitting
├── web/
│   ├── app/                    # FastAPI backend
│   │   ├── main.py             # App factory, startup/shutdown
│   │   ├── broadcast.py        # Frame capture → process → WebSocket push
│   │   ├── camera.py           # Camera detection, preview, hardware binding
│   │   ├── processor.py        # Pupil detection, eye center, 3D gaze
│   │   ├── state.py            # Tracker registry, settings, shared state
│   │   ├── persistence.py      # JSON config save/load
│   │   └── routers/            # REST + WebSocket endpoints
│   └── frontend/               # React SPA
│       └── src/
│           ├── components/     # VideoFeed, EyeModel3D, CalibrationWizard, etc.
│           ├── hooks/          # useWebSocket, useTrackingData, useTheme, useLatency
│           ├── lib/            # Calibration math, audio feedback, backend URL
│           └── types/          # TypeScript interfaces
└── pyproject.toml              # Python project config

---

🔧 Tech Stack

Layer	Tech
Vision	OpenCV, NumPy
Backend	FastAPI, Uvicorn, wsproto
Frontend	React 19, TypeScript, Tailwind CSS v4
3D	Three.js, React Three Fiber
Animation	Framer Motion
Tooling	pnpm, Biome, Ruff, uv

---

🎯 How It Works

1. Capture — Background threads grab frames from IR cameras at up to 120 fps
2. Detect — Cascaded thresholding finds the darkest region, fits an ellipse to the pupil
3. Filter — Confidence scoring, aspect ratio checks, and bounds calibration reject blinks and noise
4. Track — Eye center estimated via ellipse normal intersections (Classic or EWMA algorithm)
5. Calibrate — 9-point polynomial regression maps pupil position to screen coordinates
6. Fuse — Dual-eye gaze weighted by confidence × (1 / calibration_error)
7. Stream — Annotated frames + tracking data pushed over WebSocket at target FPS
8. Render — React renders live feeds, 3D model, metrics, heatmap, and gaze cursor

---

🖥 Supported Platforms

Platform	Camera Backend	Status
macOS	AVFoundation	Fully tested
Windows	DirectShow	Supported
Linux	V4L2	Supported

Camera detection on macOS uses system_profiler to avoid triggering iPhone Continuity Camera.

---

🙏 Credits

Pupil detection algorithm adapted from JEOresearch/EyeTracker by Jason Orlosky.