Introducing Superpipe Studio: a free and open-source tool to curate datasets, run evals, conduct experiments and optimize your LLM pipelines for accuracy, speed and cost. We've spent the last year building with LLMs and working closely with a number of companies and encountered the same problems repeatedly: - evaluating LLM software is hard, because... - collecting high-quality, representative, correcty-labeled data is hard... - which makes it hard to objectively compare different techniques, models and parameters... - which prevents continuous iteration and improvement of LLM software If you want to build a winning AI product - it all comes down to continuous evaluation and optimization. Unlike traditional software, you can't build it once, write some unit tests and sleep well at night. You need a virtuous cycle where product usage generates data that goes into your eval system, which is used to evaluate new models/techniques or fine-tune your own model. We tried existing tools in the market and found none of them really worked well to implement the virtuous cycle we needed, so we built a lightweight tool for internal use: Superpipe Studio. It helps you do 3 things: 1. curate and manage datasets 2. run experiments and compare them on accuracy/speed/cost 3. monitor your production pipelines (and expand your datasets with production data) We think everyone should have these abilities, so we're making Studio completely free and open-source with no restrictions. Studio is a work in progress and rough around the edges but still robust. We encourage you to try it and modify it to suit your needs. The gold-standard of AI software is Tesla/Waymo's full-self driving. Not every AI system needs 5 9's of accuracy, but the process for making your AI system better is the same that Tesla and Waymo followed. And it all starts with high-quality labeled data and a robust evaluation and optimization system. (Github and docs in comment)
Aman Dhesi’s Post
More Relevant Posts
-
Building AI pipelines **that work** isn't rocket science but you have to look at your data and you must experiment. We built Superpipe Studio to make this easy for ourselves and we're excited to open-source it so that everyone can build powerful AI pipelines.
Introducing Superpipe Studio: a free and open-source tool to curate datasets, run evals, conduct experiments and optimize your LLM pipelines for accuracy, speed and cost. We've spent the last year building with LLMs and working closely with a number of companies and encountered the same problems repeatedly: - evaluating LLM software is hard, because... - collecting high-quality, representative, correcty-labeled data is hard... - which makes it hard to objectively compare different techniques, models and parameters... - which prevents continuous iteration and improvement of LLM software If you want to build a winning AI product - it all comes down to continuous evaluation and optimization. Unlike traditional software, you can't build it once, write some unit tests and sleep well at night. You need a virtuous cycle where product usage generates data that goes into your eval system, which is used to evaluate new models/techniques or fine-tune your own model. We tried existing tools in the market and found none of them really worked well to implement the virtuous cycle we needed, so we built a lightweight tool for internal use: Superpipe Studio. It helps you do 3 things: 1. curate and manage datasets 2. run experiments and compare them on accuracy/speed/cost 3. monitor your production pipelines (and expand your datasets with production data) We think everyone should have these abilities, so we're making Studio completely free and open-source with no restrictions. Studio is a work in progress and rough around the edges but still robust. We encourage you to try it and modify it to suit your needs. The gold-standard of AI software is Tesla/Waymo's full-self driving. Not every AI system needs 5 9's of accuracy, but the process for making your AI system better is the same that Tesla and Waymo followed. And it all starts with high-quality labeled data and a robust evaluation and optimization system. (Github and docs in comment)
-
-
Tune Studio is an end-to-end platform for developing applications using Large Language Models. So far, I haven't seen any other platform like this one. You can do everything here: 1. You can curate your data. 2. Use the playground to play with different models and try your ideas. 3. Fine-tune an open-source model on your data. 4. Deploy the model when you are done. This is awesome for anyone building generative AI applications. You can use Tune Studio to work with any of the open-source models out there. They were one of the few companies to host Llama 2 and Llama 3 before anyone else. Here is a link to check it out: https://lnkd.in/esN8s35A One of their main selling points is that Tune Studio scales! You don't have to worry about serving your model to lots of users. They also have built-in user management, authentication, on-prem support, user context management, and pretty much everything you need to build generative AI applications. Thanks to the Tune team for collaborating with me on this post. We are living through the best years of development tools for AI developers. The field is unstoppable.
-
Building Low-code/No-code Platform to help people build better software. Generative AI | LLM | AI Agents
👋 Tech enthusiasts, AI developers, and professionals in multi-agent systems! ⚡ You NEED to hear about AutoGen Studio from Microsoft Research! AutoGen Studio is a game-changer in the world of AI development. Imagine being able to create, debug, and evaluate multi-agent systems with a 𝗱𝗿𝗮𝗴-𝗮𝗻𝗱-𝗱𝗿𝗼𝗽 interface that makes everything as easy as pie 🍰. 👨💻 The 𝗱𝗲𝗰𝗹𝗮𝗿𝗮𝘁𝗶𝘃𝗲 𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 feature empowers you to define your system’s behavior without diving into convoluted code. 🔧 𝗥𝗲𝘂𝘀𝗮𝗯𝗹𝗲 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 mean you can leverage existing functionalities, saving you tons of time and effort. 📊 And get this—𝗮𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗽𝗿𝗼𝗳𝗶𝗹𝗶𝗻𝗴 tools provide you with deep insights into your system's performance, ensuring everything runs smoothly and efficiently. But here’s the kicker: AutoGen Studio is RAPIDLY being adopted across the industry. This isn’t just a tool; it’s a revolution in how we approach AI development. 🚀💡 If you’re looking to elevate your AI projects and stay ahead of the curve, you need AutoGen Studio in your toolkit RIGHT NOW! #TechInnovation #AIDevelopment #MultiAgentSystems #LowCodeRevolution #MicrosoftResearch Let’s get to work, legends. 💪
-
Product Announcement: Introducing Cleanlab Studio's Auto-Labeling Agent! Is your team bogged down by the tedious task of manual data labeling? Are your algorithms struggling with limited examples? Cleanlab Studio’s Auto-Labeling Agent is designed to alleviate these challenges by efficiently suggesting accurate new labels to complete your dataset effortlessly. 🔍 Why It Matters: Fully automated annotation can overlook important nuances, while fully manual annotation is both error-prone and labor-intensive. By blending human and automated efforts, our Auto-Labeling Agent enhances the accuracy and efficiency of data annotation, making the process seamless and significantly quicker. These AI-suggested labels mean humans only need to focus on the rows where their manual efforts have the highest ROI. ⚙️ How It Works: Simply import a dataset with less than 50% labels, and our Auto-Labeling Agent will automatically provide high-confidence suggestions for the remaining rows. This approach allows for streamlined and rapid iterations while keeping you in full control. Our pilot users have experienced an 80% reduction in time spent on labeling and iterations. Ready to put your annotation on cruise control? 🏎️💨 Read our blog for more details and sign up for Cleanlab Studio today – it’s free to try, with no code required. Learn more: https://lnkd.in/gfQqtjm9 Sign up for Cleanlab Studio: https://app.cleanlab.ai/
-
🔬Our team put Microsoft's Florence-2 to the test, and here's our take: Florence-2 is a game-changing vision foundation model that's living up to the hype. After thorough testing, we're impressed by its versatility and performance across various computer vision tasks. Key findings from our assessment: ✅ Exceptional captioning: From basic to highly detailed image descriptions, Florence-2 delivers impressive results. 🎯 Accurate object detection: Consistently identifies and localizes objects with high precision. 🔎 Powerful segmentation: The large model excels at referring expression segmentation, though fine-tuned versions may underperform in this area. 📝 Solid OCR capabilities: Handles both printed and handwritten text, with occasional confusion between similar characters. ⚡ Performance insights: The large model (770M parameters) generally outperforms other versions across tasks. Task completion times vary, with complex tasks like segmentation taking longer (up to ~50s on M2 Pro). 💡 Pro tip: We found the base model sometimes inconsistent with multiple object detection, while the larger model showed more reliability. Florence-2 is a powerful tool for computer vision tasks, but like any AI, it has its quirks. Proper model selection and task-specific tuning are key to maximizing its potential. Have you experimented with Florence-2 or similar vision models? Tell us about it in comments. You can read more about our team and get in touch with them here: https://hubs.ly/Q02GYGyy0 #AITesting #ComputerVision #MachineLearning #TechInnovation #Microsoft
-
While you were watching the "I Do's" on Love Is Blind last night, I made a snake video game in under 20 minutes using AI... Here’s the catch: I don’t code. And yet, 141 lines of Python later, here I am with a playable game—something I used to think only belonged to the world of developers and tech pros. But thanks to LLMs and tools like Replit, the barriers that used to keep people like you and me out? They’re crumbling. Fast. You see, we’re entering a new era where you don’t need to be an expert to build something amazing. Here’s why that matters: 1. The World is Open to Everyone ↳ Whether you’re a coder, designer, or have never touched a line of code in your life—AI is your co-pilot. You can build, create, and solve problems you never thought you had the skills for. 2. Speed is the New Advantage ↳ I didn’t spend weeks learning to code or breaking my brain on tutorials. In 20 minutes, AI turned an idea into reality. This kind of speed gives you the power to experiment, iterate, and create value at an unprecedented rate. 3. You Don’t Have to Know Everything ↳ Gone are the days of gatekeeping knowledge. Whether it’s building a website, writing a business plan, or coding a game—AI is breaking down the complexity. All you need is an idea, and AI takes care of the heavy lifting. This isn’t just about a recreating the greatest addition to Nokia's software ever. This is about democratization. YouTube started the democratization of content. AI is democratizing creation. The doors that were once shut—requiring a degree or years of experience—are now wide open for anyone bold enough to step through. And, I predict that we're rapidly moving from founders with an idea and a pitch deck to the era of rapid prototyping. The future belongs to the creators who realize they don’t need to have all the answers. They just need to ask the right questions.
-
If you’re a Product Leader or Founder you need to be using AI IDEs in your workflow to quickly code proof of concepts. I've started playing with tools like Replit, v0 by Vercel, and Cursor, and am seriously impressed. In the past week, I built and deployed two simple apps—both born from casual “Wouldn’t it be useful if I had X?” moments. The best way to stay on top of technology is to use it. The speed of technological progress the past year has been insane, and it's only going to accelerate. I know many (myself included) feel there’s not enough time to explore all the new AI tools launching seemingly every day. In the coming weeks, I’ll share some of the projects I’m working on and the AI apps I’m finding most useful. Would love to hear other tools people are finding useful - for design, coding, productivity, etc..
-
Generative AI is the future, but getting started can feel like a hurdle. That's where Amazon Bedrock Studio comes in handy. In my latest blog, I check out this new playground that simplifies prototyping generative AI apps. You get access to Knowledge Bases, Guardrails, Function calling, and more - all without heavy coding required. The way I see it, we all need to start getting familiar with generative AI concepts. And Bedrock Studio provides an approachable way to experiment and learn the fundamentals. Now, being a preview, there are some quirks around access that I cover transparently. But if you want to quickly get hands-on with the latest gen AI tools for prototyping, this is a solid option. #GenerativeAI #RapidPrototyping #AmazonBedrock #TechBlogging
-
🚀 Generative AI Meets Chrome DevTools: A Game-Changer for Developers! 💻 Imagine debugging code, analyzing performance, or even crafting scripts directly in Chrome DevTools, all powered by Generative AI. This is no longer a dream—it’s the new reality reshaping how we code and innovate. Generative AI integration into Chrome DevTools offers developers an unprecedented level of assistance: ✅ Code suggestions on the fly: AI can analyze your project and recommend solutions, saving hours of debugging. ✅ Performance optimization insights: Beyond identifying bottlenecks, the AI provides actionable recommendations to streamline your apps. ✅ Real-time learning: Whether you’re a junior developer or a seasoned expert, AI-powered tools guide you with best practices, ensuring clean and efficient code. What does this mean for developers? More creativity, less grunt work! 🛠️ Now, we can focus on solving complex problems and building innovative products, while the AI takes care of repetitive tasks. Are you ready to explore this cutting-edge synergy of Generative AI and Chrome DevTools? What’s your take on this evolution? Share your thoughts! 🗣️ #AIRevolution #ChromeDevTools #GenerativeAI #TechInnovation #WebDevelopment #SoftwareEngineering #FutureOfWork #CodeOptimization #ProductivityBoost #DevToolsMagic
-
-
🛠️ Building a Witty AI Financial Advisor (Cleo-Inspired) in Less Than 3 Hours Like many others, I’ve been playing around with AI coding tools like V0, Cursor, and Replit—and yeah, they’re pretty impressive. While they’re not quite there yet for building fully shippable products, they’re amazing for spinning up quick MVPs for new product or feature ideas. Plus, working with them is super fun. As the "cost of experimentation" drops, it will be interesting to see what this means for the wider venture ecosystem—especially for fintech products. ___ The quick (and basic) prototype in the video shows an app that can ingest transactional data (from your debit/credit card) in .csv or .pdf format, analyse it, and offer financial insights via a chatbot in a witty way— similar to Cleo’s Roast Mode feature. I've built it with: Figma → V0 → Cursor → Replit in a very short time with only pretty basic coding skills.
Something new - hiring founding engineers
8moGithub: https://github.com/villagecomputing/studio