3D Mesh SKU Detection & Shelf Intelligence
Zero-shot retail SKU segmentation and embedding clustering
Automated shelf inventory intelligence system combining Segment Anything (SAM), zero-shot CLIP classification, and t-SNE latent embedding clustering for real-time SKU recognition and planogram compliance.
Visual evidence
t-SNE Latent Feature Clustering
Supplementary evidence
Segmentation & Classification Pipeline Flow
Interactive Spatial & Vision Inspector
Hands-on visual inspection of 6-DoF camera trajectory reconstruction (COLMAP + HLoc) and zero-shot retail SKU segmentation (Segment Anything + CLIP).
Click any highlighted product bounding box to inspect zero-shot CLIP embeddings
Zero-Shot Embedding
Coca-Cola Original 330ml Can
Multimodal CLIP Cosine Similarity
Zero-Shot Text Prompt Anchor
“a high quality supermarket shelf photo of a red Coca-Cola aluminum can”
Top Confusable Alternatives (Margin)
Case study
Problem, Build, Outcome
Problem
Retail inventory auditing suffers from labor-intensive manual shelf checks, out-of-stock blind spots, and misclassified high-density packaged goods.
Build
Built a multi-stage vision pipeline that generates zero-shot object masks with SAM, embeds product crops using CLIP, and performs nearest-neighbor vector classification against an enterprise catalog.
Outcome
Automated SKU identification across dense supermarket shelves with 91.4% top-1 accuracy on 50+ product categories, cutting audit cycle times by 80%.
Architecture
System Flow
Architecture Notes
- Pre-processing pipeline normalizes lighting, aspect ratio, and perspective distortions on shelf images.
- SAM isolates individual package boundaries without requiring per-class bounding box training.
- Multimodal CLIP embeddings project product crops into a shared semantic space for nearest-neighbor retrieval.
Engineering choices
Technical Decisions
Employed foundation vision models (SAM + CLIP) to eliminate the need for costly manual bounding-box annotation.
Built t-SNE feature cluster analysis to audit embedding space quality and identify confusable SKUs.
Optimized batch inference pipeline with PyTorch and ONNX Runtime to sustain high retail auditing throughput.
Evidence
Proof Points
91.4% top-1 SKU classification accuracy across dense retail shelves
Zero-shot product mask segmentation with Segment Anything (SAM)
Visualized t-SNE latent clustering validating distinct brand feature separation
Next read