AR Smart Mirror
Real-time interactive AR virtual try-on system designed for retail fashion.
The Problem
In-store clothing fitting is slow, creates long queues in fitting rooms, and increases garment wear. Retailers need an instant, hygienic, and engaging digital mirror that lets shoppers visualize outfits without undressing.
My Role
Built solo with AI-assisted development — computer vision pipeline design, MediaPipe pose detection integration, Canvas AR overlay engine, FastAPI backend services, Docker multi-container topology, and Cloudflare Zero Trust ingress.
Highlights
- Sub-30ms real-time body tracking and landmark extraction using MediaPipe Pose
- Dynamic Canvas AR garment warping and affine transformation for realistic clothing drape
- FastAPI backend with PostgreSQL catalog and Redis session pub/sub synchronization
- Mobile controller pairing allowing customers to browse catalogs and trigger try-ons from their smartphones
- Production-grade Docker Compose architecture with Nginx reverse proxy and Cloudflare Tunnel
Stack
Constraints
- Hard real-time latency target (<33ms per frame / 30 FPS) to avoid motion sickness and visual lag.
- Zero customer biometric persistence — video feeds must process in memory without saving raw footage to disk.
- Compatibility with standard consumer hardware (e.g. RTX 3060) without requiring specialized depth cameras.
System Architecture
Key Trade-offs
The decisions worth defending — what I chose, what I turned down, and why.
Pose tracking execution model
Chose
Client-side MediaPipe Pose on Canvas API with GPU acceleration
Rejected
Server-side video streaming and frame-by-frame inference
Server inference introduces network jitter, high bandwidth overhead, and privacy concerns streaming raw camera feeds over the network.
Garment deformation approach
Chose
Real-time 2D affine mesh deformation keyed to skeletal anchor landmarks
Rejected
Full 3D physics-based volumetric cloth simulation
3D cloth simulation cannot sustain 30+ FPS on edge kiosk hardware; 2D mesh triangulation provides fluid response with low computational cost.
Remote store interconnect
Chose
Cloudflare Tunnel over outbound HTTPS
Rejected
Direct port forwarding and dynamic DNS on store routers
Retail store Wi-Fi and ISP firewalls prevent inbound port forwarding; Cloudflare Tunnels punch through NAT securely without public IPs.
What I'd Do Differently
An honest retrospective — the stuff I'd change with more time, more users, or a second pass.
- 1Implement WebGPU shaders for cloth lighting and shadow blending to improve realism under dynamic store lighting.
- 2Add an offline fallback queue for store analytics so telemetry buffers locally during internet interruptions.
- 3Optimize landmark smoothing filters (OneEuro filter) to eliminate micro-jitter when shoppers stand at edge distances.
Computer Vision & Pose Tracking Shell
Benchmark the real-time AR try-on pipeline (status, camera init, pose detect, mesh align -shirt, latency bench).
Technical Deep-Dive
Architecture, specifications, and implementation details.
03 — System Architecture
#Layer Overview
┌───────────────────────────────────┐ ┌───────────────────────────────────┐
│ MIRROR DISPLAY (Client A) │ │ MOBILE CONTROLLER (Client B) │
│ Webcam · MediaPipe · Canvas AR │ │ Phone WebApp · Garment Catalog │
└─────────────────┬─────────────────┘ └─────────────────┬─────────────────┘
│ HTTPS / WSS │ HTTPS / WSS
┌─────────────────▼────────────────────────────────────▼─────────────────────┐
│ CLOUDFLARE TUNNEL │
│ DNS · SSL · Zero Trust · Any WiFi worldwide │
└───────────────────────────────────┬────────────────────────────────────────┘
│
┌───────────────────────────────────▼────────────────────────────────────────┐
│ NGINX REVERSE PROXY (Docker: ar-frontend) │
│ SSL Termination · Static files (Mirror & Mobile) · Rate limiting │
└───────────────────────────────────┬────────────────────────────────────────┘
│
┌───────────────────────────────────▼────────────────────────────────────────┐
│ FASTAPI GATEWAY (Docker: ar-api :8000) │
│ REST /api/v1/* · WebSocket Gateway (Session Sync) · JWT Validation │
└──────┬───────────────┬──────────────┬───────────────┬──────────────────────┘
│ │ │ │
┌──────▼──┐ ┌───────▼──┐ ┌───────▼─────┐ ┌───────▼──────┐
│ Pose │ │ Garment │ │ Analytics │ │ WebSocket │
│ Service │ │ Service │ │ Service │ │ Pub/Sub │
│ :8001 │ │ (in-api) │ │ (in-api) │ │ (Redis sync) │
└──────┬──┘ └───────┬──┘ └──────┬──────┘ └───────┬──────┘
│ │ │ │
┌──────▼──┐ ┌───────▼──┐ ┌──────▼──────┐
│ Pose │ │ Garment │ │ Analytics │
│ Service │ │ Service │ │ Service │
│ :8001 │ │ (in-api) │ │ (in-api) │
└──────┬──┘ └───────┬──┘ └──────┬──────┘
│ │ │
┌──────▼───────────────▼──────────────▼───────────────┐
│ DATA LAYER │
│ PostgreSQL 15 · Redis 7 · Volume Storage │
│ Prometheus + Grafana (monitoring) │
└─────────────────────────────────────────────────────┘
#Docker Services
| Container | Image | Port | Role |
|---|---|---|---|
| ar-frontend | nginx:alpine | 80/443 | Static + proxy |
| ar-api | python:3.11-slim | 8000 | FastAPI |
| ar-pose | python:3.11+cuda | 8001 | MediaPipe |
| ar-db | postgres:15-alpine | 5432 | Database |
| ar-cache | redis:7-alpine | 6379 | Cache |
| ar-monitor | grafana/grafana | 3000 | Metrics |
#Hardware
- CPU: Intel i5 14th Gen
- GPU: NVIDIA RTX 3060 12GB (MediaPipe + CUDA)
- RAM: 16GB DDR4
- Storage: 1TB HDD
#Network Flow (Session Pairing)
1. Mirror starts session -> Generates QR Code
2. User Phone scans QR -> Opens Cloudflare URL
3. Phone & Mirror both connect to WebSocket Gateway
4. Phone sends "select_garment" event -> FastAPI WebSocket
5. FastAPI broadcasts to Mirror -> Mirror renders garment