Spotify data assistant Vedder processes 13,000 conversations using human-curated clusters
Spotify Engineering details the architecture of "Vedder," their internal AI data assistant, which helps employees query petabytes of data across 70,000 datasets by generating SQL queries. The system relies on human-curated data domain "clusters" and explicit expert review of training examples to ensure accuracy and trustworthiness, with experts accepting only 12.5% of automatically proposed question-SQL pairs. Vedder has been used by over 2,100 Spotify employees in over 13,000 conversations since August 2025.
Key Takeaways
- Vedder has handled over 60,000 messages across 177 distinct data clusters including advertising, podcasts, and finance.
- Experts rejected 87.5% of AI-proposed SQL training pairs, highlighting that raw query history contains excessive noise.
- The system supports a 25% non-technical user base through Slack, IDE servers, and a dedicated web UI.
- A continuous health score identifies when schema changes or table deprecations degrade the assistant's accuracy.
Why It Matters
Internal tool scalability is the primary bottleneck for data-driven streaming giants handling 70,000+ datasets. By shifting data scientists from answering Slack queries to curating AI context layers, Spotify is commoditizing domain expertise. This "expert-in-the-loop" model addresses the trust deficit inherent in large-scale LLM deployments where raw schemas lack business nuance. As streaming platforms face increasing pressure to optimize ad yields and subscriber retention, the ability to democratize data access without sacrificing precision becomes a structural competitive advantage. If successful, this architecture sets a new standard for how media organizations bridge the gap between petabyte-scale warehouses and non-technical business decision-makers. Watch for whether Spotify open-sources the 'Cluster' curation framework to its Backstage ecosystem.
Additional Context
The launch of Vedder follows a period of intense internal engineering transformation at Spotify. During its Q4 2025 earnings call in February 2026, co-CEO Gustav Söderström revealed that the company’s push into generative AI extends to its core development pipeline. The company’s internal deployment system, 'Honk,' reportedly allows engineers to use Anthropic's Claude Code to ship features and bug fixes directly from mobile devices. Söderström noted that in late 2025, some of Spotify's most senior developers had not written a single line of manual code for months, serving instead as supervisors for AI agents. This shift contributed to a record 50 major updates delivered to the consumer app in 2025 alone, including AI-driven 'Prompted Playlists.' Financially, Spotify’s heavy R&D investment is showing results. Per financial reports from May 2026, the company surpassed 761 million monthly active users, including nearly 300 million premium subscribers. Executives at the June 2026 Investor Day highlighted that the company’s technology now analyzes trillions of daily interactions to power its 'super-premium' tier, which reportedly includes AI-based song customization developed via a partnership with Universal Music Group. These internal gains come at a time when the broader streaming analytics market is projected to reach $131 billion by 2034, according to IMARC Group reporting from early 2026. Spotify’s Chief Architect, Niklas Gustavsson, emphasized in a June 2026 technical presentation that the company’s long-term strategy involves making developer and data experiences 'agent-ready.' By standardizing 70,000 datasets into a unified software catalog within its Backstage platform, Spotify ensures that AI tools like Vedder and Honk encounter predictable environments. This standardization has led to a 76% increase in pull request frequency, as reported in June 2026, positioning the company to maintain its lead in a market where content and infrastructure are increasingly governed by real-time automated insights.
Read full article at engineering.atspotify.com
Get this in your inbox → Subscribe
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source