Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models
Research paper by Ally Qin and collaborators. arXiv record dated 2026-03-27. Research area: Machine learning.
Resource information
- Authors
- Ally Qin, Jian Wan, Sarat Mudunuri, Srinivasan Manoharan
- Research area
- Machine learning
- Application domain
- Agent systems
- AI methods
- Large language models
- Research topics
- cs.LG, cs.AI
- arXiv record date
- 2026-03-27
- Publication
- arXiv
- Upstream last modified
- Apr 23, 2026
- License
- CC BY 4.0
- Last record update
- Nov 4, 2026
Abstract
We evaluate speculative decoding with EAGLE3 as an inference-time optimization for PayPal's Commerce Agent, powered by a fine-tuned llama3.1-nemotron-nano-8B-v1 model. Building on prior work (NEMO-4-PAYPAL) that reduced latency and cost through domain-specific fine-tuning, we benchmark EAGLE3 via vLLM against NVIDIA NIM on identical 2xH100 hardware across 40 configurations spanning speculative token counts (gamma=3, gamma=5), concurrency levels (1-32), and sampling temperatures (0, 0.5). Key findings: (1) gamma=3 achieves 22-49% throughput improvement and 18-33% latency reduction at zero additional hardware cost; (2) acceptance rates remain stable at approximately 35.5% for gamma=3 across all conditions; (3) gamma=5 yields diminishing returns (approximately 25% acceptance rate); (4) LLM-as-Judge evaluation confirms fully preserved output quality; and (5) speculative decoding on a single H100 matches or exceeds NIM on two H100s, enabling 50% GPU cost reduction.