Contrastive learning is a core component of modern dense retrieval systems, but its effectiveness heavily relies on the quality of negative examples used during training. While hard negative mining (HNM) consistently improves offline metrics, a fundamental question remains: do offline gains translate to improved user experiences and business outcomes?
- Core Problem Definition
- Method: Structured Negative Sampling + LLM Relevance Evaluation
- Training Data Generation Pipeline
- Data Mixtures
- Model Architecture
- Training Configuration
- Evaluation Framework
- Experimental Results
Negative Data Mining for Contrastive Learning in Dense Retrieval at IKEA.com
Core Problem Definition
Contrastive learning is a core component of modern dense retrieval systems, but its effectiveness heavily relies on the quality of negative examples used during training. While hard negative mining (HNM) consistently improves offline metrics, a fundamental question remains: do offline gains translate to improved user experiences and business outcomes?
Research Questions: - RQ1: How do different hard negative mining strategies affect dense retrieval performance for product search? - RQ2: What factors contribute to the generalization gap between synthetic and real user queries? - RQ3: How do offline retrieval metrics correlate with online user engagement and purchase intent?
Formal Statement: Given a query q, document collection D, and contrastive loss L, hard negative mining selects negatives N_hard ⊂ D that are semantically similar to q but irrelevant, as opposed to random negatives N_random. The question is whether the offline metric improvement Δ_offline(N_hard) predicts online metric improvement Δ_online(N_hard).
Method: Structured Negative Sampling + LLM Relevance Evaluation
Training Data Generation Pipeline
Stage 1: Category Feature Extraction - LLM extracts category-specific attributes from product catalog (373 leaf categories) - For each category: sample products with names, descriptions, common search terms, metadata - LLM identifies visual (colors), material, size, activity-based, and user-need attributes - Ensures category-appropriate attributes (e.g., "solid wood" for furniture, not textiles)
Stage 2: Synthetic Query Generation - LLM generates 50 search queries per category across complexity tiers: - Simple (30%): 1-2 words (e.g., "grey sofa") - Medium (35%): 3-4 words combining attributes (e.g., "grey velvet corner sofa") - Complex (35%): 5+ words (e.g., "small grey corner sofa bed with storage") - Also generates intent-based: activity-based, situational, product-name queries - Total: 6,300 unique queries
Stage 3: Training Dataset Labeling - LLM scores query-product relevance on 1-5 scale - Positives: score ≥ 4; Negatives: score ≤ 2; Score 3 excluded for clear separation - Produces 448,073 triplets classified into: - Attribute-only (244K): Same category, different attribute (e.g., query "white sofa", negative is black sofa) - Cross-category (105K): Matching attribute, wrong category (e.g., query "white sofa", negative is white table) - Multi-attribute (93K): Queries with multiple attributes where each negative violates exactly one
Data Mixtures
| Mixture | Triplets | Description |
|---|---|---|
| Baseline | 7.2M | Random negative sampling |
| HNM | 7.9M | Baseline + 448K LLM-labeled hard negatives |
| HNM + QE | 8.3M | HNM + 450K query expansion triplets |
| HNM + QE Full | 8.8M | HNM + QE + 897K exhaustive expansion |
Model Architecture
Late-interaction retrieval model (ColBERT-style):
S(q,d) = Σ_{i∈|E_q|} max_{j∈|E_d|} E_{q_i} · E_{d_j}^T
where E_q and E_d are contextualized token embeddings for query and document.
Training Configuration
- Learning rate: 3 × 10^{-6}
- Batch size: 32
- Max document length: 512 tokens
- Max training steps: 500,000 with early stopping on validation loss
- Hardware: NVIDIA A100 GPUs, ~18 hours per configuration
- Statistical robustness: 5 random seeds per configuration, report mean ± std
Evaluation Framework
Synthetic Query Benchmark: 70 LLM-generated evaluation queries (held-out categories), stratified by complexity Real Query Benchmark: 100 queries from production search logs (35 single-attribute, 35 multi-attribute, 30 product-name) Online A/B Test: 2-week production deployment, 50/50 traffic split, focused on long-tail queries (≥3 words)
Category Accuracy@K (Cat@K):
Cat@K = |{p ∈ Retrieved@K : cat(p) ∈ C_q}| / K
where C_q is the set of leaf categories from relevant ground truth products.
Experimental Results
Offline Evaluation
| Model | Synthetic R@10 | Synthetic Cat@10 | Synthetic Cat@50 | Real R@10 | Real Cat@10 | Real Cat@50 |
|---|---|---|---|---|---|---|
| Baseline | 44.9±0.1 | 76.5±0.6 | 47.5±0.1 | 41.2±0.9 | 73.4±0.4 | 62.1±0.4 |
| +HNM | 49.1±0.4 | 80.8±0.3 | 48.7±0.2 | 41.2±0.4 | 76.0±0.7 | 64.6±0.3 |
| +HNM+QE | 48.2±0.5 | 79.8±1.0 | 48.1±0.5 | 38.3±0.9 | 76.0±1.0 | 64.2±0.3 |
| +HNM+QE Full | 47.6±0.4 | 79.8±0.4 | 48.9±0.5 | 39.8±0.9 | 74.1±0.7 | 63.6±0.3 |
Key findings: 1. HNM improves Cat@10 by +4.3% on synthetic queries but R@10 on real queries remains unchanged 2. QE variants do not outperform HNM despite 900K more triplets — data quality > data quantity 3. Real queries span 7x more categories than synthetic (12.5 vs 1.7), explaining the generalization gap
Online A/B Test Results
No statistically significant differences between HNM and baseline across all user engagement metrics (click-through rate, add-to-cart rate, search interaction rate; p > 0.05).
The +2.6% average category accuracy improvement observed offline did not translate to measurable changes in user behavior.
User Behavior Analysis Explaining the Gap
- 44.7% of queries are broad category searches where baseline already performs well
- 67% of popular searches exhibit zero-click rates above 50% — users frequently satisfy needs from search results page without clicking
- When majority of sessions end without clicks regardless of ranking quality, retrieval improvements have limited room to influence engagement metrics
- Long-tail query restriction (≥3 words) limited statistical power
Comparison with Related Work
| Study | Offline Gain | Online Gain | Key Finding |
|---|---|---|---|
| Amazon (Wang et al. 2023) | Various | Up to 97% directional agreement | NDCG discriminative power >99% |
| AliExpress (Huzhang et al. 2021) | Yes | Inconsistent | Context effects matter |
| Walmart (Lin et al. 2024) | Yes | Needed extra relevance filtering | False positives invisible offline |
| IKEA (this work) | +2.6% Cat@10 | None (p>0.05) | Zero-click behavior explains gap |
Critical Analysis
Strengths
- Honest reporting of negative results: The A/B test showed no improvement, and the paper thoroughly explains why — this is valuable for the community
- Rigorous three-stage evaluation: Synthetic → Real → Online provides a complete picture
- Quantified the zero-click problem: 67% zero-click rate on popular searches is a critical insight for e-commerce search
- Identified the intent breadth gap: Real queries span 7x more categories than synthetic — this explains why HNM on narrow synthetic queries doesn't generalize
- Data quality > quantity finding: HNM+QE Full (8.8M) underperforms HNM (7.9M), a clear demonstration that more data isn't always better
Weaknesses / Questions
- LLM label quality: Ground truth is LLM-generated (1-5 scale), validated by 93% human agreement. But LLM labels may have systematic biases (leniency) that affect both training and evaluation.
- Limited online test scope: Only long-tail queries (≥3 words), 2-week duration. Short-head queries where most revenue occurs were excluded.
- Single market: Only Canada market tested. Cross-market generalization unknown.
- No interleaving experiment: Standard A/B tests lack sensitivity for ranking improvements compared to interleaving (Bi et al. 2022).
- Single architecture: Only late-interaction model tested. Results may not generalize to bi-encoder or cross-encoder architectures.
- No analysis of position bias: Offline evaluation ignores position bias which is known to affect online metrics.
Implications for Our Work
- The offline-online gap is a critical lesson: improving embedding quality in isolation may not improve end-user metrics
- Zero-click behavior means click-based metrics are insufficient — need alternative signals (scroll depth, wishlist, filter interactions)
- The intent breadth gap (synthetic vs real queries) suggests training data should match the actual query distribution
- For retrieval systems, this work is a cautionary tale: always validate offline improvements with online experiments
- The structured negative sampling approach (taxonomy-aware) is sound but needs to be paired with real-query distribution matching