Case Study

AR Smart Mirror

Real-time interactive AR virtual try-on system designed for retail fashion.

The Problem

In-store clothing fitting is slow, creates long queues in fitting rooms, and increases garment wear. Retailers need an instant, hygienic, and engaging digital mirror that lets shoppers visualize outfits without undressing.

My Role

Built solo with AI-assisted development — computer vision pipeline design, MediaPipe pose detection integration, Canvas AR overlay engine, FastAPI backend services, Docker multi-container topology, and Cloudflare Zero Trust ingress.

Official Website ↗Mobile Controller ↗Architecture Docs ↗Try-On Pipeline ↗

Highlights

Sub-30ms real-time body tracking and landmark extraction using MediaPipe Pose
Dynamic Canvas AR garment warping and affine transformation for realistic clothing drape
FastAPI backend with PostgreSQL catalog and Redis session pub/sub synchronization
Mobile controller pairing allowing customers to browse catalogs and trigger try-ons from their smartphones
Production-grade Docker Compose architecture with Nginx reverse proxy and Cloudflare Tunnel

Stack

Python 3.11FastAPIMediaPipeHTML5 CanvasPostgreSQL 15Redis 7DockerNginx

Constraints

Hard real-time latency target (<33ms per frame / 30 FPS) to avoid motion sickness and visual lag.
Zero customer biometric persistence — video feeds must process in memory without saving raw footage to disk.
Compatibility with standard consumer hardware (e.g. RTX 3060) without requiring specialized depth cameras.

System Architecture

Capture & Client

Webcam Feed

MediaPipe Pose (Client A)

Mobile Controller (Client B)

Edge & Ingress

Cloudflare Tunnel

Nginx Reverse Proxy

SSL Termination

Application Services

FastAPI Gateway (:8000)

Garment Overlay Service

WebSocket Pub/Sub

Data & Storage

PostgreSQL 15 (Catalog/Users)

Redis 7 (Session Sync)

Local Volume Storage

Telemetry

Prometheus

Grafana Monitoring (:3000)

Key Trade-offs

The decisions worth defending — what I chose, what I turned down, and why.

Pose tracking execution model

Chose

Client-side MediaPipe Pose on Canvas API with GPU acceleration

Rejected

Server-side video streaming and frame-by-frame inference

Server inference introduces network jitter, high bandwidth overhead, and privacy concerns streaming raw camera feeds over the network.

Garment deformation approach

Chose

Real-time 2D affine mesh deformation keyed to skeletal anchor landmarks

Rejected

Full 3D physics-based volumetric cloth simulation

3D cloth simulation cannot sustain 30+ FPS on edge kiosk hardware; 2D mesh triangulation provides fluid response with low computational cost.

Remote store interconnect

Chose

Cloudflare Tunnel over outbound HTTPS

Rejected

Direct port forwarding and dynamic DNS on store routers

Retail store Wi-Fi and ISP firewalls prevent inbound port forwarding; Cloudflare Tunnels punch through NAT securely without public IPs.

What I'd Do Differently

An honest retrospective — the stuff I'd change with more time, more users, or a second pass.

1Implement WebGPU shaders for cloth lighting and shadow blending to improve realism under dynamic store lighting.
2Add an offline fallback queue for store analytics so telemetry buffers locally during internet interruptions.
3Optimize landmark smoothing filters (OneEuro filter) to eliminate micro-jitter when shoppers stand at edge distances.

Computer Vision & Pose Tracking Shell

Benchmark the real-time AR try-on pipeline (status, camera init, pose detect, mesh align -shirt, latency bench).

icelegends-interactive-shell ~ [armirror]

History: ↑ / ↓

cv-engine@ar-mirror:~/pipeline$

Technical Deep-Dive

Architecture, specifications, and implementation details.

09 — Garment Processing Pipeline

"

For the full multi-stream try-on architecture (3D cloth reconstruction, semantic segmentation, skin-part compositing) see 19_TryOn_Pipeline_Architecture.md.

#Upload Pipeline (backend)

When a garment photo is submitted via /admin.html → POST /api/v1/garments:

User uploads photo (any format, any background)
        │
        ▼
[Step 1] rembg — U2Net background removal
        │  → transparent RGBA PNG
        ▼
[Step 2] Trim transparent margins
        │  → bounding box of opaque pixels + 2% padding
        ▼
[Step 3] Fit inside 800 × 860 canvas (aspect-ratio preserving)
        │  → centred on transparent canvas
        ▼
Saved as /app/garments/{name}_nobg.png
Served at /garments/{name}_nobg.png

##Target canvas: 800 × 860 px

Aspect ratio ≈ 0.93:1 — matches shoulder-to-hip torso coverage on a standing adult. The pipeline normalises any input (portrait, landscape, flat-lay, ghost mannequin).

#Real-Time Rendering Pipeline (frontend — three_engine.js v61)

Once the mobile controller selects a garment, the mirror renders it every camera frame:

MediaPipe onResults (30 fps)
        │
        ├─ results.poseLandmarks  (33 keypoints)
        │        │
        │        ▼
        │   Extract key landmarks
        │     nose [0], shoulders [11,12], hips [23,24]
        │        │
        │        ▼
        │   Compute 4 garment corners (canvas pixel space)
        │     collarY  = shoulderMidY − (noseToShoulder × 0.65)
        │     hemY     = hipMidY      + (bodyLen × 0.30)
        │     tl = { x: leftShoulderX  − collarSpan×0.25, y: collarY }
        │     tr = { x: rightShoulderX + collarSpan×0.25, y: collarY }
        │     bl = { x: leftHipX       − hipSpan×0.40,    y: hemY   }
        │     br = { x: rightHipX      + hipSpan×0.40,    y: hemY   }
        │        │
        │        ▼
        │   LERP smooth corners
        │     collar corners: factor 0.18  (snappy)
        │     hem corners:    factor 0.10  (slow, reduces jitter)
        │        │
        │        ▼
        │   Bilinear interpolation → 289 mesh vertices (16×16 grid)
        │     for each (u, v) in [0,1]²:
        │       cx = bilinear(tl.x, tr.x, bl.x, br.x, u, v)
        │       cy = bilinear(tl.y, tr.y, bl.y, br.y, u, v)
        │       vertex.setXYZ(idx, cx, H − cy, 0)   // flip Y for Three.js
        │        │
        │        ▼
        │   Three.js renderer.render(scene, camera)
        │     → garment mesh drawn on #webglCanvas (WebGL, transparent bg)
        │     → #cvOv (2D canvas) shows video below
        └─ composite: video | WebGL garment

##Why a 16×16 subdivided mesh?

A flat drawImage rectangle can only scale uniformly. The subdivided mesh (289 vertices) allows each row and column to track independently: shoulders can be wider than hips, the garment tilts with body lean, and perspective foreshortening is approximated — all without any ML inference beyond the existing MediaPipe pose.

#Garment Type Routing

Garment type (DB field)	Render slot	Engine
tshirt, tee, hoodie, jacket, shirt, sweater, coat, dress	`top`	Three.js WebGL quad-mesh
pants, shorts, skirt, jeans, chinos, joggers	`bottom`	Canvas 2D drawImage + hip clip
sunglasses	`accessories`	Canvas 2D drawImage at ear landmarks

#Landmark Index Reference

Index	Landmark	Used for
0	Nose	Collar height calculation
11	Left shoulder	Collar-left corner X
12	Right shoulder	Collar-right corner X
23	Left hip	Hem-left corner X
24	Right hip	Hem-right corner X

#PNG Requirements for Uploaded Garments

Property	Requirement
Format	PNG or JPG (any — pipeline converts to PNG)
Background	Any (rembg removes it automatically)
Min resolution	400 × 400 px
Ideal shot	Flat-lay or ghost mannequin, front-facing
Shoulders	Visible at top 15–30 % of image

Do not manually remove the background before uploading — the pipeline handles it and will produce better edges than manual cutouts.

~ End of Document ~