Pareto AI

Today we’re releasing AttuneBench v1.0: the first open EQ benchmark built from real multi-turn human-AI conversations and scored against what the person in the conversation actually felt and wanted. Most existing EQ benchmarks rely on synthetic prompts, single-turn interactions, or third-party ratings. But emotional intelligence is private knowledge. Only the person in the conversation knows whether they felt understood, what kind of response they wanted, and whether the interaction actually helped. So instead of asking outside annotators to judge conversations, we asked the participants themselves. We evaluated 11 frontier models across 200 real conversations and 50,000+ participant annotations. One result stood out immediately: Models were much better at recognizing how another model responded than predicting what the participant actually wanted from the conversation. For example, a model could correctly identify that a response was: - validating - analytical - reassuring - advice-oriented while still failing to predict whether the participant actually wanted reassurance, analysis, or advice in that moment. Several frontier models also ranked responses opposite to participant preference more often than with it. In other words: sounding emotionally fluent is not the same thing as understanding what a person actually needs. We also found that emotional understanding dropped substantially for participants reporting mental health diagnoses, even while models remained relatively strong at identifying surface conversational patterns. AttuneBench v1.0 is open-source and free to run, and we’ll continue updating it as models and methods evolve. Paper, leaderboard, code, and blog post are linked in the comments. Huge credit to Kate Lubrano and Mark Whiting on the Pareto AI research team, Karina Nguyen and the Thoughtful team, and especially the Pareto annotators who lived and labeled these conversations firsthand.

2,302 people. 22 would have received harmful medical advice. Zero actually did. AI models are giving medical and mental health advice to millions of people. Can you prevent harmful advice by adding safety instructions to the prompt? The UK's AI Security Institute recently tested this. They deployed the same chatbot twice: once with minimal safety prompting, once with explicit safety instructions. The finding: safety prompts had no meaningful impact on harmful advice rates. What did work? A classifier trained on expert-labeled data to detect harmful outputs in real-time. AISI brought in my team at Pareto to build the training dataset. We recruited licensed doctors, therapists, and career coaches and coached them to decompose complex professional judgment into verifiable steps. Together we developed harm grading rubrics and built 6,707 evaluated examples. Fine-tuning Llama 8B on this data boosted accuracy at detecting harmful advice from 77% to 96%, beating GPT-4o's zero-shot performance (93%). Real-world impact: AISI deployed this classifier in a study with 2,302 participants. Without the safety layer, 22 people would have received harmful advice. With it? Zero harmful messages delivered. The key insight: You genuinely can't prompt your way to safety in domains requiring professional judgment and deep contextual understanding. For high-stakes domains like medical, mental health, and career advice, expert supervision creates meaningfully better outcomes than instruction-based approaches alone. The methodology that scales: 1. Bring experts in from day one to co-design 2. Build workflows that elicit professional judgment 3. Capture reasoning and context, not just labels AISI open-sourced everything: paper, model, and dataset. At Pareto, we're building systems that make it easy for frontier experts to contribute sustainably to AI training. The future isn't about replacing human expertise, it's about building better systems to capture expert insight at the edge of what's known. Deep gratitude to Elizabeth Nguyen and Daria Butuc at Pareto.AI, and Lennart Luettgau and Henry Davidson at UK's AI Security Institute for making this collaboration succeed. #ArtificialIntelligence #AIResearch #AIEthics

It’s a special kind of full-circle moment when you get to walk into the very first SF office of a founder you’ve supported since the very beginning as an investor…AND later had the joy of collaborating with at Slope 💜 Huge congratulations to the team at Pareto.AI on their officewarming and revealing the new brand experience last night!! 🥂I love my bucket hat! Over the last year, Slope partnered with the Pareto team on everything from brand strategy and positioning (“Humanity to infinity ✨”), to a bold new identity and website. I especially love our art direction, which pairs calming futurism with human-centered imagery - balancing expansive, nature-forward visuals with meaningful interactions between people and technology. Case study and shoutouts to the Slope team behind the incredible work coming soon...but for now, congratulations to the Pareto team!! 🙌 So proud Phoebe Yao!

In this new episode, Steven and I sat down with Phoebe Yao, founder and CEO of Pareto.AI, a human data platform that helps Anthropic, Google DeepMind, Character.AI and many other frontier labs source their expert data for LLM training and evals. Before founding Pareto, Phoebe was a Thiel Fellow and a Human Centered Design & Engineering major at Stanford. We discussed with Phoebe her experience with Stanford ASES, Launchpad, Lean Launchpad at d.school, what made her decide to drop out, how she survived through pivots and many more! Some of my takeaways: • Finding pmf is an ongoing challenge in fast-moving markets. • Pivots aren’t selling your soul. They’re about meeting market needs while staying true to your mission to keep the team motivated. • Learn not to defer decisions. Clear, confident calls are the foundation of strong leadership. • Nurture relationships. Business is about people wanting to work with you. The link to the full episode is in the comments section👇  (And check out their brand new website pareto.ai!)