Anthony Alcaraz

How fast is fast when it comes to commercial open source software traction? Here are some stats to serve as benchmarks that Matthieu Lavergne put together among many other great data and insights in Serena's first report on commercial open source software. It's available for free here: https://lnkd.in/d5uBvjAU

18

3 commentaires

Our AI research team at Martian is advancing one of the most important areas of AI: Mechanistic Interpretability. This involves understanding how models make decisions by examining their internal mechanisms, similar to looking inside a clock to see how it works. Our research has long-term implications for AI, with safety being a primary consideration. By understanding the inner workings of AI models, we can better predict and mitigate potential risks, ensuring more reliable and secure AI systems. We are putting the outputs of this research to work today in our LLM Router, which predicts the best large language model (LLM) for a given prompt based on a customer’s cost, latency, and output quality preferences. #AI #MachineLearning #ArtificialIntelligence #TechInnovation #MechanisticInterpretability #Research #MartianAI #AITransparency

20

3 commentaires

When to Invest in Long Context In-Context Learning ? 🗼 As language models continue to grow in size and capability, the way we use them is also evolving. A recent study by Bertsch et al. explores the potential of in-context learning (ICL) with long-context models, pushing the boundaries of how many examples can be provided as demonstrations. The results are surprising and thought-provoking, suggesting that in certain data regimes, long-context ICL could be a powerful alternative to traditional fine-tuning and retrieval methods. Traditional few-shot ICL, where only a handful of examples are given, has already seen wide adoption due to its simplicity and reasonable performance. However, it can fall short when dealing with complex tasks or large label spaces. On the other hand, fine-tuning the entire model on a large dataset has been the go-to approach for maximal performance, but comes with higher training costs and reduced flexibility. Long-context ICL offers a middle ground, allowing the model to attend to many more relevant examples without expensive fine-tuning. The study highlights that as the number of demonstrations grows, the importance of careful example selection diminishes, making it possible to reuse a single set of examples for all inputs. It is prudent to already state that LLMops system design will be a careful balance between retrieval with few shot examples (with knowledge graph or vector database), in context learning with long context model, and fine tuning to handle reasoning and context.

27

"A team of computer scientists and AI researchers from FAIR at Meta, INRIA, Université Paris Saclay and Google, has developed a possible means for automating data curation for self-supervised pre-training of AI datasets. The group has written a paper describing their development process, the technique they developed and how well it has worked thus far during testing. It is posted on the arXiv preprint server. As developers and users alike have been learning over the past year, the quality of the data that is used to train AI systems is tied very closely to the accuracy of results. Currently, the best results are obtained with systems that use manually curated data and the worst are obtained from systems that are uncurated." #datacuration #aidatasets

2

You will find here the latest HAI report on 2024 trends: https://lnkd.in/e2VHzaNJ As always, this is very deep, fact based and insightful. Thanks Nicolas Marescaux for sharing! Here are the key take aways: 1. AI beats humans on some tasks, but not on all. AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics and planning. 2. Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable machine learning models, while academia contributed only 15. 3. Frontier models get way more expensive. According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute. 4. The United States leads China, the EU, and the U.K. as the leading source of top AI models. In 2023, 61 notable AI models originated from U.S.-based institutions, far outpacing the European Union’s 21 and China’s 15. 5. Robust and standardized evaluations for LLM responsibility are seriously lacking. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks. 6. Generative AI investment skyrockets. Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds. 7. The data is in: AI makes workers more productive and leads to higher quality work. In 2023, several studies assessed AI’s impact on labor, suggesting that AI enables workers to complete tasks more quickly and to improve the quality of their output. 8. Scientific progress accelerates even further, thanks to AI. In 2022, AI began to advance scientific discovery. 2023, however, saw the launch of even more significant science-related AI applications— from AlphaDev, which makes algorithmic sorting more efficient, to GNoME, which facilitates the process of materials discovery. 9. The number of AI regulations in the United States sharply increases. The number of AI- related regulations in the U.S. has risen significantly in the past year and over the last five years. Last year alone, the total number of AI-related regulations grew by 56.3%. 10. People across the globe are more cognizant of AI’s potential impact—and more nervous. A survey from Ipsos shows that, over the last year, the proportion of those who think AI will dramatically affect their lives in the next three to five years has increased from 60% to 66%. Moreover, 52% express nervousness toward AI products and services, marking a 13 percentage point rise from 2022.

26

2 commentaires

At Toloka, we’re not just connecting data contributors with AI projects—we’re shaping the future of AI architecture. By designing pipelines that merge human expertise with synthetic data, we aim to enhance and solidify AI models. Read this article from our CEO, Olga Megorskaya, on this pioneering approach. #AI #DataArchitecture #SyntheticData #LLMs #Toloka #genAI #LLMs

1

1 commentaire

Another quick note on evaluations and benchmarks for agentic workflows. As I mentioned in the article I shared yesterday (link in comments), agentic systems are still not very reliable. However, increasingly robust benchmark proposals are emerging in various verticals: 𝑻𝒐𝒐𝒍 𝒖𝒔𝒂𝒈𝒆: Here, WorkBench stands out, which tests an LLM’s ability to use 26 software tools to operate on five simulated workplace databases: email, calendar, web analytics, projects, and customer relationship management. Tools include deleting emails, looking up calendar events, creating graphs, and looking up tasks in a to-do list. 𝑺𝒐𝒄𝒊𝒂𝒍: SocialDial is a benchmark that tests potential social violations which, for example, may depend on cultural context. Anecdotally, in China, reserving places with many occurrences of the number 4 is considered bad luck. Similarly, there is CharacterEval for role performance. 𝑷𝒍𝒂𝒏𝒏𝒊𝒏𝒈: Google and colleagues propose a planning benchmark called Natural Plan that includes thousands of prompts as benchmarks in categories such as travel planning, virtual calendar schedules, and physical location schedules, where the metric to optimize is to make more people fit in a place at one specific time. These tests have unambiguous responses, providing a good set of tests for an agentic workflow. 🔗 WorkBench: https://lnkd.in/dUyynJTx 🔗 Natural Plan: https://lnkd.in/dJNVwkXZ 🔗 In addition to those mentioned, this is a good repository with references to benchmarks for agentic workflows: https://lnkd.in/d5V3JNeB ---- #AI #AgenticSystems #Automation #Innovation #LLM #Agents #multiagents #AI

4

3 commentaires

🚀🔍 A great analysis by Index Ventures on Vertical AI companies as part of the Application Layer of the AI stack. "With the AI Platform Shift upon us, we believe that the next logical iteration of vertical SaaS will be Vertical AI – vertically-focused AI platforms, bundled alongside workflow SaaS, built on top of models which have been uniquely trained on industry-specific datasets." The few things that will differentiate the winners in the race, as mentioned in the article:  - access to proprietary industry data,  - effectively train large language models against those datasets,  - package those models via applications,  - deliver tremendous utility with fast time-to-value for customers, At Neurolabs, we care a lot about the time-to-value, which is why we’ve built these exact steps for our Image recognition solution.✅ With speed of execution & user experience at core, we go a step further and train our large vision models with synthetic data, delivering an even faster onboarding experience for our CPG customers & partners. 🤝 If you're facing challenges with your IR projects, let’s talk. There's no need to rely on outdated methods. Synthetic IR is here to propel us into the future! 🚀 #syntheticimagerecognition #syntheticdata #computervision #imagerecognition #cpg #retailexecution  

16

1 commentaire

Sometimes I'm asked whether agents will shape the future... And I give an example, #ChatGPT is already an agent-driven workflow. Ask him ! ChatGPT from #OpenAI integrates tools (like Python Environment), other AI models (like DALL-E), memory, and multi-step tasks, likely using GPT to manage different system messages that assume various "personas.", in other words multi-simple-agent flows. It's important to recognize that agents and agent-driven workflows are already a part of our daily lives. Therefore, I recommend that enterprises explore and stay informed about the research trends concerning agent-driven workflows and multi-agent systems. To build a community for sharing information about agent-driven workflows and multi-agent systems, myself and other enthusiasts are launching AIgentic! 👥 Spread the Word! Help us reach more innovators like you. Share our journey, and let's unlock Agent's full potential together! 🌟 Elevate Your AI Agents Insights with AIgentic 🌟 #ArtificialIntelligence #Agents #GenerativeAI

11

2 commentaires

💥 New day, new model 💥 We ran our performance benchmarks on Anthropic Claude 3.5 Sonnet v/s OpenAI's GPT-4o & the results weren't what you'd expect ⏬ Anthropic says Claude 3.5 Sonnet outperforms all competitor models on key evaluations, at twice the speed of Claude 3 Opus and one-fifth the cost. In our benchmarks 3.5 Sonnet & 4o came out neck & neck. Ultimately, you need to run evaluations on your task at hand and measure the quality yourself. 🔍 Data Extraction: Both models identified 60-80% of data correctly, but neither excelled in this task. 📊 Classification: Claude 3.5 Sonnet (72%) outperformed GPT-4o (65%) in mean accuracy. However, GPT-4o led in precision (86.21%), crucial for accurately classifying customer tickets. 🧠 Verbal Reasoning: GPT-4o outperformed Claude 3.5 Sonnet with 69% accuracy, versus 44%. Both models did well on analogy and relationship questions. 📜 Context window: Claude 3.5 Sonnet boasts a larger context window (200K) compared to GPT-4o's 128K 🌪 Latency: Claude 3.5 Sonnet is 2x faster than Claude 3 Opus, but it’s still lags behind GPT-4o when it comes to latency (50% faster Time to First Token) Price and throughput for these models was very similar Want to learn more about how these models stack up? Check out our full analysis on the Vellum blog: https://lnkd.in/dwEcQJe4 Oh and did I say, Claude 3.5 Sonnet is already in Vellum? Sign up for a demo to run your evals & start building!

68

1 commentaire

What will you do in your first 90 days as head of data? We’ve gathered top data experts to share their secrets to success at the Day1 Summit. Join Mehdi Ouazza Tiankai Feng, Timo Dechau, Vladimir Lagutinskiy, Christophe Blefari, and Ust Oldfield as they share insights on the do’s and don’ts of the first 90 days. They’ve been where you are and are here to help you navigate this critical period. Why you can’t miss Day1 summit: ✨ Get a curated action plan and playbook for success in your new data role ✨ Gain practical, battle-tested advice from data leaders at top companies ✨ Network and connect with some of the smartest minds in data  Secure Your Spot Now: https://lnkd.in/g9xZdvYG

24

1 commentaire

Proud to see Pathway join the OPEA platform by The Linux Foundation AI & Data! It’s inspiring to witness how GenAI leaders are coming together to advance the frontier of GenAI and RAG technologies. This collaboration perfectly aligns with our mission to deliver enterprise-grade AI solutions with real-time data synchronization. Let's push the boundaries of what's possible in AI, together. 💪 #GenAI #LLM #OpenSource #EnterpriseAI

31

🌟 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐭𝐡𝐞 𝐅𝐮𝐬𝐢𝐨𝐧 𝐨𝐟 𝐆𝐫𝐚𝐩𝐡 𝐓𝐡𝐞𝐨𝐫𝐲 𝐚𝐧𝐝 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 🌟 During my first year of my master’s program (M1), I dove into several fascinating topics, with a strong focus on graph theory. This exploration led me to a powerful intersection: 𝐆𝐫𝐚𝐩𝐡 𝐍𝐞𝐮𝐫𝐚𝐥 𝐍𝐞𝐭𝐰𝐨𝐫𝐤𝐬 (𝐆𝐍𝐍𝐬). Inspired by this, I wrote a Medium article titled "𝗚𝗿𝗮𝗽𝗵 𝗡𝗲𝘂𝗿𝗮𝗹 𝗡𝗲𝘁𝘄𝗼𝗿𝗸𝘀: 𝗧𝗵𝗲 𝗕𝗿𝗮𝗶𝗻𝗰𝗵𝗶𝗹𝗱 𝗼𝗳 𝗚𝗿𝗮𝗽𝗵 𝗧𝗵𝗲𝗼𝗿𝘆 𝗮𝗻𝗱 𝗗𝗲𝗲𝗽 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴",  where I explain how GNNs work and their underlying principles. 📌 You can find the article here: 𝘩𝘵𝘵𝘱𝘴://𝘮𝘦𝘥𝘪𝘶𝘮.𝘤𝘰𝘮/@𝘢𝘣𝘥𝘦𝘭𝘩𝘦𝘥𝘪.𝘺𝘰𝘶𝘴𝘴𝘦𝘧8/𝘨𝘳𝘢𝘱𝘩-𝘯𝘦𝘶𝘳𝘢𝘭-𝘯𝘦𝘵𝘸𝘰𝘳𝘬𝘴-𝘵𝘩𝘦-𝘣𝘳𝘢𝘪𝘯𝘤𝘩𝘪𝘭𝘥-𝘰𝘧-𝘨𝘳𝘢𝘱𝘩-𝘵𝘩𝘦𝘰𝘳𝘺-𝘢𝘯𝘥-𝘥𝘦𝘦𝘱-𝘭𝘦𝘢𝘳𝘯𝘪𝘯𝘨-3𝘤83𝘥6𝘢233𝘥7 Feel free to give it a read and share your thoughts ! #𝙂𝙧𝙖𝙥𝙝𝙏𝙝𝙚𝙤𝙧𝙮 #𝘿𝙚𝙚𝙥𝙇𝙚𝙖𝙧𝙣𝙞𝙣𝙜 #𝙂𝙧𝙖𝙥𝙝𝙉𝙚𝙪𝙧𝙖𝙡𝙉𝙚𝙩𝙬𝙤𝙧𝙠𝙨 #𝘼𝙄 #𝙈𝙖𝙘𝙝𝙞𝙣𝙚𝙇𝙚𝙖𝙧𝙣𝙞𝙣𝙜

23

Exciting #news for the #generativeai field! Today #antrophic announced: 👉 an upgraded Claude 3.5 Sonnet. 👉 a new model: Claude 3.5 #haiku which matches the performance of Claude 3 Opus on many evaluations for the same cost and similar speed to the previous generation of Haiku. 👏 a groundbreaking #computeruse capability which allows people to direct Claude to use computers the way we do: i.e. by looking at a screen, moving a cursor, clicking buttons, and typing text. Whilst at this stage this is still experimental, it is great to see this new capability being released early for #feedback from developers to further improve the model's performance. If you have a couple of minutes, have a look at the #computeruse demo: it is truly impressive!

5

Presenting Orpomis Prime 7B model series ! These models are a result of trying different merge strategies of the following models: - mistral-orpo-beta from KAIST AI - Hermes-2-Pro-Mistral-7B from NousResearch In my experiments, dare ties merge leads to a significantly better-performing model on the openllm benchmarks In general, the model performance is quite solid in text generation as well as math reasoning. Full result comparison: https://lnkd.in/gb92DXwG Dare ties Model: https://lnkd.in/gMS-H4Sq Model stock Model: https://lnkd.in/giwEJ7Gs

9

Meta has released Llama 3 (backed by a $10 billion investment) and is available to public for FREE 😲 . What could be the business strategy behind it 🤔 ? It could probably: 1.     Position Meta as a leader in open source. 2.     Create potential hurdles for smaller AI developers. 3.     Make it harder for companies like OpenAI to charge a premium. 4.     Drive adoption of its platforms by integrating with Instagram, Whatsapp, and Facebook (be ready for lots of Adsssss and suggestionssss….) It’s worth remembering the old wisdom: If the product is free, you might be the product! 🙄 #Meta #Llama3 #AI #OpenSource #BusinessStrategy #TechnologyNews

17

2 commentaires

#agentic #llms #future #ai If it's not obvious, it should be.. there are other ways to collect data and process it then the internet. Direct connections to the observable real world directly to train new software constructs will be an emerging field now that's now that the simulations have gone pretty well. Rhis is powerful computing of the future. While we turn our eyes to building applications with narrow AI today, this next round of robots will be created in parallel. The LeCunn Wave will have very different operations and results that few humans will understand but many humans will gain the benefit from. #futuresthinking

1

🚀 𝐔𝐧𝐯𝐞𝐢𝐥𝐢𝐧𝐠 𝐭𝐡𝐞 𝐌𝐚𝐠𝐢𝐜 𝐨𝐟 𝐅𝐢𝐧𝐞-𝐓𝐮𝐧𝐢𝐧𝐠 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 (𝐋𝐋𝐌𝐬) 🎯 🔗 𝐑𝐞𝐚𝐝 𝐭𝐡𝐞 𝐟𝐮𝐥𝐥 𝐬𝐭𝐨𝐫𝐲 𝐨𝐧 𝐌𝐞𝐝𝐢𝐮𝐦: From Generic to Genius: The Art of Fine-Tuning Large Language Models Have you ever wondered how Large Language Models (LLMs) like GPT or BERT transform from being generalists into specialists for tasks like medical diagnostics, sentiment analysis, or legal insights? 𝐈𝐧 𝐦𝐲 𝐥𝐚𝐭𝐞𝐬𝐭 𝐌𝐞𝐝𝐢𝐮𝐦 𝐚𝐫𝐭𝐢𝐜𝐥𝐞, I take you on a storytelling journey through the art of fine-tuning LLMs. Discover: ✅ 𝐖𝐡𝐚𝐭 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐢𝐬 𝐚𝐧𝐝 𝐰𝐡𝐲 𝐢𝐭’𝐬 𝐚 𝐠𝐚𝐦𝐞-𝐜𝐡𝐚𝐧𝐠𝐞𝐫. ✅ 𝐇𝐨𝐰 𝐭𝐨 𝐩𝐫𝐞𝐩𝐚𝐫𝐞 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬 𝐚𝐧𝐝 𝐢𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐞 𝐜𝐮𝐬𝐭𝐨𝐦 𝐜𝐥𝐚𝐬𝐬𝐢𝐟𝐢𝐜𝐚𝐭𝐢𝐨𝐧 𝐡𝐞𝐚𝐝𝐬. ✅ 𝐒𝐭𝐞𝐩-𝐛𝐲-𝐬𝐭𝐞𝐩 𝐜𝐨𝐝𝐞 𝐬𝐧𝐢𝐩𝐩𝐞𝐭𝐬 𝐟𝐨𝐫 𝐥𝐨𝐚𝐝𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥𝐬, 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠, 𝐚𝐧𝐝 𝐭𝐞𝐬𝐭𝐢𝐧𝐠. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐭𝐢𝐩𝐬 𝐭𝐨 𝐭𝐮𝐫𝐧 𝐠𝐞𝐧𝐞𝐫𝐚𝐥-𝐩𝐮𝐫𝐩𝐨𝐬𝐞 𝐋𝐋𝐌𝐬 𝐢𝐧𝐭𝐨 𝐭𝐚𝐬𝐤-𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐠𝐞𝐧𝐢𝐮𝐬𝐞𝐬. ✨ Fine-tuning isn't just about adjusting models; it's about unlocking their full potential for real-world problem-solving. 👩‍💻 Whether you're a 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫, 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐫, 𝐨𝐫 𝐞𝐧𝐭𝐡𝐮𝐬𝐢𝐚𝐬𝐭, this guide simplifies the technical steps and presents them as an engaging story. 🔍 𝐓𝐚𝐤𝐞 𝐚 𝐥𝐨𝐨𝐤 𝐚𝐧𝐝 𝐬𝐡𝐚𝐫𝐞 𝐲𝐨𝐮𝐫 𝐭𝐡𝐨𝐮𝐠𝐡𝐭𝐬: 📖 𝐑𝐞𝐚𝐝 𝐭𝐡𝐞 𝐚𝐫𝐭𝐢𝐜𝐥𝐞: https://lnkd.in/g2wEW2gr 🔄 Repost to inspire others in your network. 💬 Let’s discuss your fine-tuning experiences or challenges in the comments! #LLM #FineTuning #GenerativeAI #MachineLearning #AIForGood #TheLLMInsider

29