19 — Try-On Pipeline Architecture

#Overview

The AR Mirror try-on system processes three independent inputs in parallel and fuses them into a single photorealistic try-on result. The three streams are:

┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│ Reference Body   │     │  Target Cloth    │     │  Target Human    │
│ Model (3D mesh)  │     │  (product photo) │     │  (live camera)   │
└────────┬─────────┘     └────────┬─────────┘     └────────┬─────────┘
         │                        │                         │
         ▼                        │                         ▼
┌──────────────────┐              │              ┌──────────────────────┐
│ Reference Mask   │◄─────────────┘              │ Semantic Segmentation│
│ Generation       │                             │ (body-part labelling)│
└────────┬─────────┘                             └──────────┬───────────┘
         │ Ref. Mask                                        │ Human
         ▼                                                  │ Segmentation
┌──────────────────┐                                        │
│  2D Cloth        │◄── Target Cloth ──────────────────────►│
│  Matching        │                                        │
└────────┬─────────┘                             ┌──────────▼───────────┐
         │ Cloth (2D matched)                    │ 2D Pose Estimation   │
         ▼                                       │ (landmark detection) │
┌──────────────────┐                             └──────────┬───────────┘
│  3D Cloth        │                                        │ 2D joints
│  Reconstruction  │                             ┌──────────▼───────────┐
└────────┬─────────┘                             │ 3D Pose & Shape      │
         │ 3D Cloth Model                        │ Estimation           │
         │                                       └──────────┬───────────┘
         │                                                  │ 3D pose & shape
         │                              ┌───────────────────┤
         ▼                              ▼                   ▼
┌──────────────────────────┐  ┌──────────────────┐  ┌──────────────────────┐
│  3D Shape Transfer &     │  │ Target Segment.  │  │ Target Skin Parts    │
│  Re-posing               │  │ Generation       │  │ Generation           │
│  (cloth warped to body)  │  └────────┬─────────┘  └──────────┬───────────┘
└────────────┬─────────────┘           │ Target seg.            │ Skin parts
             │ 3D warped cloth         │                        │
             └─────────────────────────┴────────────────────────┘
                                       │
                             ┌─────────▼─────────┐
                             │   Try-On Fusion    │
                             │   (compositing)    │
                             └─────────┬─────────┘
                                       │
                             ┌─────────▼─────────┐
                             │   Try-On Result   │
                             └───────────────────┘

#Stage-by-Stage Breakdown

##Stream A — 3D Cloth Model

Stage A1: Reference Mask Generation

• Input: Reference body model (parametric 3D mesh, e.g. SMPL)
• Output: Ref. Mask — a silhouette image encoding body-part regions
• Purpose: Provides a canonical shape template so the cloth matching stage knows where shoulders, torso, and hips are in normalised body space.

Stage A2: 2D Cloth Matching

• Input: Ref. Mask + Target Cloth photo (product image)
• Output: Cloth (2D matched) — the garment warped onto the reference silhouette
• Purpose: Aligns the flat garment photo with the reference body outline so that seams, collar, and hem land on the correct body-part boundaries.

Stage A3: 3D Cloth Reconstruction

• Input: 2D matched cloth + reference body geometry
• Output: 3D Cloth Model — a textured mesh of the garment draped on the reference body
• Purpose: Lifts the 2D aligned garment into 3D so it can be re-posed to match the target person's actual body pose and shape.

##Stream B — Target Human Analysis

Stage B1: 2D Pose Estimation

• Input: Target human frame (live camera or static image)
• Output: 2D joints — a set of 2D keypoints (skeleton landmarks)
• Current implementation: MediaPipe Pose, 33-point model, 30 fps
• Key landmarks used: shoulders [11,12], hips [23,24], nose [0]

Stage B2: 3D Pose & Shape Estimation

• Input: 2D joints + human image context
• Output: 3D pose & shape — a full SMPL-family body mesh fitted to the person
• Purpose: Recovers the person's body volume (not just 2D silhouette), enabling the cloth to wrap correctly around shoulders, chest depth, and waist.
• Current implementation: Approximated by bilinear quad-mesh deformation anchored to 4 body-derived control points (collar-left, collar-right, hem-left, hem-right).

##Stream C — Semantic Understanding

Stage C1: Semantic Segmentation

• Input: Target human frame
• Output: Human Segmentation — per-pixel body-part labels (head, torso, left-arm, right-arm, legs, background, etc.)
• Current implementation: MediaPipe segmentation mask (binary person/background)

Stage C2: Target Segmentation Generation

• Input: Human Segmentation + 3D pose & shape
• Output: Target Segmentation — refined pixel mask aligned to the predicted 3D body
• Purpose: Produces precise garment region boundaries (where cloth should start/end) accounting for body pose, so the fusion stage doesn't bleed cloth onto the face or hands.

Stage C3: Target Skin Parts Generation

• Input: Target Segmentation
• Output: Target Skin Parts — isolated face, neck, hand regions (RGBA patches)
• Purpose: These skin patches are composited on top of the warped cloth in the fusion stage so that hands and face always appear in front of the garment.

##Fusion — Try-On Fusion

Stage F1: 3D Shape Transfer & Re-posing

• Input: 3D Cloth Model (Stream A) + 3D pose & shape (Stream B)
• Output: 3D Warped Cloth — the reference cloth re-posed to the target person's body
• Purpose: Physically warps the cloth mesh to match the target's skeletal pose and body shape, producing correct fold deformations, sleeve length, and torso width.

Stage F2: Try-On Fusion

• Input: 3D Warped Cloth + Target Segmentation + Target Skin Parts
• Output: Try-On Result — final composite frame
• Steps:
  1. Render 3D warped cloth into the camera view using the estimated camera parameters
  2. Mask the rendered cloth to the torso segmentation region
  3. Composite target skin parts (face, hands) on top of the cloth layer
  4. Blend with the original background

#Current vs. Target Implementation

#Garment Image Requirements

For the current pipeline to produce good results:

Images uploaded via /admin.html are automatically processed:

1. Background removed by rembg (U2Net model)
2. Transparent margins trimmed (2 % padding kept)
3. Resized and centred on an 800 × 860 transparent canvas

#Key Constants (three_engine.js v61)

#Comparison with nawodyaishan/ar-fashion-tryon

Reference repo: https://github.com/nawodyaishan/ar-fashion-tryon (audited at commit on main, 2026-05-15)

This section documents why the current garment looks the way it does and how the reference repo approaches the same problem, based on direct code inspection — not just the README. The repo is a final-year project with five separate services in one monorepo; AR Mirror is a single product. The comparison below maps service to service.

##Why the garment looks "wrong" today

Every garment in this project is a flat 2D PNG warped onto a 16×16 quad mesh between four MediaPipe-derived anchors (collar-left, collar-right, hem-left, hem-right). Direct consequences:

The 800 × 860 px upload canvas (see 09_Garment_Processing_Pipeline.md) bakes the "front-facing 2D PNG" assumption into the DB. Changing the visible structure of the garment therefore requires changing the asset structure first.

##Repository layout of ar-fashion-tryon (what's actually in it)

The densepose/ and detectron2/ folders inside catvton-gradio/ are vendored in full (~150 source files) — this is not a slim dependency, it's the whole Facebook AI library copy-pasted in.

##Architecture comparison

##Garment structure comparison — the key finding

Both projects represent the garment as a 2D RGBA PNG with a removed background. Neither uses a true 3D mesh, neither rigs the garment to a skeleton, neither stores per-vertex weights or cloth-physics parameters.

The reference repo's better-looking screenshots come not from a different garment structure, but from a second, offline pipeline. In CatVTON mode the flat PNG plus the person photo plus DensePose + SCHP masks are fed to a Stable-Diffusion-inpainting model, which synthesises a photo-realistic image of the person wearing the garment. That is image generation, not garment rendering — it takes seconds per frame on a GPU and is not real-time.

In its AR (real-time) mode, ar-fashion-tryon is actually less automatic than AR Mirror: the user must drag the garment onto themselves with a mouse, because there is no pose-driven mesh deformation; the only "smart" behaviour is a 15-pixel magnetic snap to the detected shoulder line.

##What this means for "I want a different garment structure"

If by "different structure" you mean the cloth wraps the body better — that is solvable inside the existing 2D PNG asset format. If you mean the asset itself is no longer a flat picture — that requires replacing both the upload pipeline and the renderer.

Four concrete paths, in increasing cost:

1. Stay 2D, add semantic regions to the existing PNG asset. Render sleeves, collar, hem, side seams as separate rules anchored to elbow [13,14], nose [0], knee [25,26] respectively. Asset stays one PNG; only the renderer changes. Fixes: sleeves, hem fall, collar tracking. Does not fix: drape, occlusion, depth. Cost: renderer rewrite only.

2. Add segmentation-based skin compositing. Implement Stage C3 (Target Skin Parts) from section above using MediaPipe Selfie Segmentation (already shipped with MediaPipe Pose, just turn enable_segmentation=True on). Composite hands/face over the garment. Asset format unchanged. Fixes: hands no longer disappear behind cloth. Cost: one extra mask blit per frame.

3. Adopt the reference repo's approach for hero shots only — run a CatVTON-style diffusion model server-side for a "save my outfit" button, keeping the real-time quad-mesh for live preview. Requires a GPU on-prem (incompatible with the current on-premises CPU constraint) or accepting a cloud dependency. The reference repo demonstrates that this is the only path to photo-realistic output today.

4. True 3D garment per item — glTF/GLB mesh with a skeleton rig per SKU, driven by MediaPipe pose. The reference repo tried this in ml-backend/previous codes/3D approch 1.py (PyTorch3D + DepthAnythingV2 to lift a single product photo into a 3D mesh) and abandoned it — the file sits in a previous codes/ quarantine. The realistic version of this requires per-garment 3D authoring, which breaks the "admin uploads a product photo" workflow.

Option 1 is the cheapest visible improvement to garment shape. Option 2 is the cheapest fix for the "hand-behind-cloth" artefact. Option 3 is the only path to photo-realism, at the cost of GPU and latency. Option 4 is structurally the "real" answer but is unproven — the reference repo's own attempt did not ship.

##Implementation log

##Realism stack (added 2026-05-15)

Five passes layered on top of the segmentation-mask base. None require new dependencies or a GPU; total added cost is ~1 ms per frame.

Why this combination matters: the reference repo nawodyaishan/ar-fashion-tryon produces realistic results only via the offline CatVTON diffusion pipeline (seconds per frame, GPU-bound). The five passes above approximate the most-visible cues that pipeline learns — silhouette-aware edges, collar occlusion, side shading, hem blending, correct front/back depth ordering — without any ML inference beyond the already-running MediaPipe Pose. They are not photo-realistic, but together they cross the threshold from "PNG taped to a body" to "cloth that responds to lighting and pose."

Tuning constants (current values, in drawGarmentCanvas2D)