Jivitesh Poojary

Interesting research paper alert : https://lnkd.in/giwG6bnZ Been thinking about evaluators and can they be self taught ! Stumbled upon this research paper and here is a Quick read about this : This paper on "Self-Taught Evaluators" introduces a novel approach to improving large language model (LLM) evaluators without relying on human-annotated data. The authors present an iterative self-improvement scheme that generates synthetic training data, allowing the LLM-as-a-Judge to self-train by generating contrasting model outputs and reasoning chains. This iterative process enables the evaluator to produce more accurate judgments, resulting in continuous performance improvement without the need for expensive and potentially outdated human annotations. Key contributions and insights from the paper include: Self-Taught Iterative Process: The core innovation is the use of synthetic data to generate contrasting model responses, which the LLM-as-a-Judge then evaluates. Over iterations, the model self-trains on its own outputs, improving its judgment accuracy. Performance Gains: The self-taught method shows impressive results, improving the Llama-3-70B-Instruct model's accuracy on RewardBench from 75.4 to 88.7, outperforming other commonly used LLM evaluators like GPT-4. Synthetic Data Creation: Synthetic preference pairs are generated without human input by prompting the LLM to produce one high-quality response and one lower-quality response to a modified version of the original instruction. This method allows the model to self-annotate its own training data. Iterative Fine-Tuning: In each iteration, the model uses the judgments it generates to improve its ability to evaluate responses. As the model's accuracy increases, the quantity of correct training examples increases as well, driving further improvement. LLM-as-a-Judge: The paper emphasizes the growing trend of using LLMs not just as response generators but also as judges, especially for open-ended or complex instructions where human evaluation is costly and subjective. The LLM-as-a-Judge provides explainable reasoning chains alongside its judgments, which enhances interpretability. Generalization: The method shows strong generalization across different domains, including reasoning, coding, and other complex tasks, highlighting its flexibility and scalability. Overall, this work demonstrates a significant step forward in LLM-based evaluation by leveraging self-training techniques and synthetic data, which reduce dependency on human annotations and accelerate model development. Saylee Muley You might want to use this, I remember you trying to use an LLM to evaluate the results from LLMs. #generativeAI #LLMs #AI

7

5 Comments

Ran a quick test on the new #OpenAI #o1mini to solve IIT-JEE Advanced 2024 maths questions. It accurately solved 13 out of 17 questions with an impressive accuracy of ~76%. Quick thoughts: 1. The hidden chain of thought is quite detailed and extensive. Breaks the problem, solves and verifies and iterates multiple times. This might increase the output token cost by a big margin. 2. Seems there are tools integrated for calculations. Much needed as #gpt4o does a bad job at it. 3. Latency is quite high. Some questions took more than 50 seconds to get answered. This needs to come down for production user-facing use cases. 4. Hallucination still exists. Check the 1st question in the link, it had the correct step-by-step reasoning but hallucinated at generating the final answer. 5. Overdoing self-thought. Saw a couple of instances where it did generate correct answers in the process but eventually made wrong answers. Would give a clearer picture once the rate limits are increased and benchmarking is done on a bigger sample. Let me know any interesting findings and thoughts 😀 https://lnkd.in/giSCezyG #GenAI #OpenAI #PhysicsWallah

139

5 Comments

🌟 How to Effectively Prepare for Data Scientist Interviews at Top FAANG Companies Early in my career, I learned that strategic preparation makes a huge difference in data science interviews. Here's what I discovered works best: 1️⃣ Research the specific team and role: Each team within big tech companies has unique needs. A data scientist on Amazon's supply chain team might need different skills than one on their recommendation systems team. Look up the team's products and recent work. 2️⃣ Master the fundamentals deeply: Top companies evaluate your understanding of core concepts like: Statistics and probability Machine learning algorithms SQL and data manipulation Experimental design Product Sense 3️⃣ Understand the 'why': Be prepared to explain: Why you chose specific approaches Trade-offs between different methods How you'd handle edge cases Business impact of your decisions In the competitive world of data science, thorough preparation focused on fundamentals and critical thinking is key to success. This approach helps you tackle any technical challenge with confidence! 💡 Looking for structured preparation? Check out Bosscoder Academy's program: ✅ Comprehensive curriculum covering ETL, Big Data, Cloud, and Advanced Data Operations ✅ Guidance from industry professionals ✅ Hands-on projects to build your portfolio Learn more:  https://bit.ly/4hYzYJO Please do your research before you enrol for any program #sponsored #datascience #data #product #amazon #google #interviews

194

3 Comments

The big plan all along was to replace human beings right? Otherwise why spend hundreds of billions on neural nets trained with effectively stolen scraped data exposing you to all kinds of liability? Here we see the plans are now more like, can it do summaries of notes if we modify long-standing human workflows?

4

Resharing it: This is a great summary shared by the LinkedIn engineering team about their #llmops and LLM deployment work and their key learnings. #llms #chatgpt4 #gemini #aiml #mlops #llmops https://lnkd.in/d2bH5GRR

2

🤖 What is Google AutoML? 🚀✨ Google AutoML takes the complexity out of machine learning by allowing anyone—even without coding experience—to create powerful AI models! 💡 Why Use Google AutoML? 1️⃣ No Coding Required 🖱️: Build AI models with a simple drag-and-drop interface. 2️⃣ Custom Models 🎯: Create solutions tailored to your data and needs. 3️⃣ Time-Saving ⏱️: Automates the hard parts of model training and optimization. 4️⃣ Accurate Results 📊: Leverages Google’s advanced machine learning infrastructure. 💡 Perfect for: Image recognition 📸 Text analysis 💬 Predictive analytics 📈 Whether you’re a beginner or an expert, Google AutoML empowers you to unlock the power of AI effortlessly. 🚀 #GoogleAutoML #AI #MachineLearning #Automation #TechInnovation #BotCampusAI

4

Awesome companies #hiring Data Scientists 1. Middesk (We make software to build trust between businesses) Hiring: Data Scientist 2. Reframe (App to help people quit or cut back on alcohol use) Hiring: Data Scientist 3. Veryfi (We Liberate End Users from Manual Data Entry.) Hiring: Data Scientist - Fraud Detection 4. Aer Compliance (Compound compliance: everything you need to keep your firm safe in a single place.) Hiring: Senior Data Scientist, Crypto 5. Netflix Hiring: Data Scientist 6. National Basketball Association (NBA) Hiring: Data Scientist, Game Analytics & Strategy 7. Twitch Hiring: Data Scientist 8. Spotify Hiring: Data Scientist II - Trust & Safety 9. SynergisticIT (The Best Programmers in the Bay Area.....Period!) Hiring: Junior Data Scientist 10. Coinbase (We're building an open financial system for the world.) Hiring: Data Scientist Follow me for more posts like this. If you're hiring and want to post on MissionHires for free, please DM me.

17

🌟 Exploring the Clear Side of AI: Why Transparency Matters In a world where artificial intelligence shapes much of our daily interactions, understanding its processes is no longer optional—it's imperative. In my latest piece on Medium, I delve into Explainable AI (XAI), the innovative field making AI's complex decisions transparent and trustworthy. This exploration highlights how XAI plays a pivotal role in improving trust, enhancing compliance, and refining operations across sectors like healthcare and financial services. 🔗 Dive into the article to see how XAI is paving the way for a future where AI’s sophistication is matched by its clarity. We’re stepping into an era where every professional can confidently interact with AI technologies. 👀 Here’s the full story: https://lnkd.in/eW-3wtc5 💬 I’m eager to hear your views! What are your thoughts on the evolution of AI transparency? Drop a comment below or send me a message to chat more about it!

10

2 Comments

Metrics are gameable and quite amusingly at times. A friend once told me what happened at a large tech giant. The bonus of Team-1 depended on improving the time complexity of the recommendation engine. The bonus of team-2 was dependent on improving the space complexity of the data fetching service. In the appraisal month, Team-1 started allocating a larger cache space to improve the time complexity and increase bonuses. Team-2 focussed on the brute force search approaches to improve the space complexity required by the algorithm. Both metrics improved for respective teams. The time and space complexity of the entire pipeline was preserved. Things went back to being as they were after bonuses were given out. Join me for a webinar on Tuesday, 26th November to understand how to select metrics that are not gameable in experimentation: https://lnkd.in/gszag6HK #metrics #experimentation #gameability

17

2 Comments

🌟 Exciting Feature Alert! 🌟 I’m thrilled to be featured in a fantastic article on BetterAuds titled “Embracing the Future of AI: Introducing Machine Learning for Product Managers”. In this insightful post, the author explores how Machine Learning is reshaping product management and why it’s crucial for product managers to embrace these technologies for better product outcomes. 🤖💡 The article also highlights my book, Machine Learning for Product Managers (Link here: https://lnkd.in/gVkczjqU ), as a key resource for anyone looking to integrate machine learning into their product development strategy. It’s an exciting time for AI and ML, and if you’re a product manager eager to stay ahead in this AI-driven landscape, I highly recommend checking it out! 📚 👉 Read the full article here: https://lnkd.in/gvkkrZtw #AI #MachineLearning #ProductManagement #TechInnovation #DataScience #ArtificialIntelligence #MLForPM #Leadership #FutureOfAI #ProductStrategy #MachineLearningForProductManagers

8

Check out this highly recommended course for those interested in optimizing their models: - Introduction to Quantization - Symmetric and Asymmetric Quantization Modes - Hands-on Implementation from Scratch using PyTorch - covering Per Tensor, Per Channel (Axis), and Per Group techniques - Weights Packing: Exploring 2 or 4 Bits Quantization Methods https://lnkd.in/gMAuWXHm

43

I am happy to share that our years of effort have finally culminated in a VLDB paper titled "Datalog with First-Class Facts." We introduce SLOG (a variant of Datalog), where facts are first-class entities, with each fact uniquely identified by an identifier. This constrained approach proves particularly efficient for reasoning about tree-structured data using Datalog, with successful applications across database systems, AI, and programming language domains. We have developed a data-parallel MPI-based backend that can scale to thousands of processes. Check out the scaling result on a supercomputer for performing Context-Sensitive Points-to-Analysis (CSPA) in our language. The collaborative work was done by Thomas Gilray, Kristopher Micinski Yihao Sun, Sowmith K, and myself. Thanks to Argonne Leadership Computing Facility for supporting our research. #database #hpc #MPI #datalog

94

1 Comment

Finding hiring ML/AI Engineers difficult? Yes + it's expensive. Here are some great start-up/scale-up stats (from Pave's Matt Schulman) from last week #freshcut ▶️ 1 in 5 engineers are getting a signing bonus when hired ▶️ 1 in 4 AI/ML professionals get a signing bonus when hired ✨ Good heads up for candidates who are looking; but also a good non-recurring cash solution to lock in that special candidate (with some retention clauses 😊 ). #signingbonuses #spotbonuses #pave #compensation #scaleups

11

4 Comments

Join me and our team from Tredence Inc., along with our partner Databricks (Sumit Saraswat & Michael Riley), for a talk on 'Unlocking Financial Insights with LLMs'. The use of #GenerativeAI in finance accelerates insights generation, resulting in cost savings, faster processes, and increased outreach for BFSI firms. By leveraging large language models (#LLMs), private equity, investment, and portfolio management firms can streamline strategies, optimize portfolio performance, and uncover market opportunities, driving greater value for investors and stakeholders. We will explore the future of financial analytics and discover how LLMs are revolutionizing the industry. Register now to secure your spot: Unlocking Financial Insights with LLMs In our session, we will cover: - AI-driven summarization for financial reports in due diligence and investment strategies - Integration with top AI models like OpenAI's GPT, Mistral AI's Mistral, Meta's Llama, and Databricks's DBRX. - Exploring Databricks’ AI capabilities, including governance, vector search, and curated open-source LLMs #FinancialInsights #LLMs #Tredence #Databricks #AI #MachineLearning #DataScience #BFSI #GenerativeAI Arpan Dasgupta Unmesh Kulkarni lakshmikant G. Ashwini Agrawal Zohra Ladha Indranil Bhattacharya Ravindra Patil, Ph.D. Aravind Chandramouli Sagar Balan

70

1 Comment

Regardless of what role you work in, there are two “musts” that everybody should be cognizant of when it comes to AI: 1. All AI/ML models produce statistical inferences, which means you can never be 100% sure the outcome will be what you expect or desire. 2. All AI/ML models are trained at a snapshot in time, meaning if an AI model works well the day after it’s trained, there’s no guarantee you’ll see continued performance down the road like a year later or so. I bring these up to highlight the importance of model validation procedures. Essentially, you want to establish a framework to make ensure you’re consistently getting the desired performance from any AI solution. And it doesn’t necessarily have to come in the form of a technical solution. It’s not ideal, but human validation can be a sufficient means of validation. Regardless of how your model validation plans manifest, you still want to ensure you have some form of plan!

26

4 Comments

Watching Gradient drop a Fortune 100's Databricks jobs costs by >50% in a few days is crazy. Everyone was shocked. 🔥 🔥 Best part was - it was all automatic. 🔥 🔥 While everyone was in meetings, building other pipelines, or just grabbing lunch - Gradient was at work dropping costs with no overhead at all. 💰 Gradient literally prints money for our customers ⚡ What's even crazier is Gradient isn't even done yet, it's still early on its journey in optimization The power of AI/ML in the cloud infrastructure world is, in my opinion, the killer app. And it's amazing to see this new technology in action helping real people. But don't take my word for it. Try it out for yourself. Book a demo here if you want help automating and managing your Databricks jobs Sync Computing

45

4 Comments

🧠 Apple Intelligence is a GPT wrapper.  💤 GPT : We will not tell what Data we used Bias: Who cares 🤷🏻‍♀️ 🤷🏻‍♂️ GUP Designers / Manufacturers : we are building better G/.. Processing Unit (PU) Data Privacy and Security will be a big question with the Apple and Open AI Engagement. No one has questioned OpenAI , Anthropic ... on biases ❓ All LLMs are biased towards the origin of the open-source data, the mindset of the authors, and people who can write/speak English and had early access to the internet. All LLMs exhibit biases stemming from the data they were trained on, which is often skewed towards Western, English-speaking sources and perspectives. This can lead to a lack of knowledge or misrepresentations about other regions, cultures, and languages, particularly those from the Global South like India and Africa. The biases in LLMs are a significant issue that needs to be addressed through more diverse and representative training data, as well as techniques like prompt engineering and human oversight. Relying solely on existing models without accounting for their limitations and biases can perpetuate harmful stereotypes and marginalize underrepresented groups. Hope Apple and Open AI collaboration bring some solutions towards. Sources for further digging: https://lnkd.in/gDAJkgei https://lnkd.in/gPYStYEi https://lnkd.in/gB6H3-wj

6

📊 Optimizing RAG Applications with TruLens: A Quick Guide Recently explored TruLens for evaluating and fine-tuning Large Language Models (LLMs) in Retrieval Augmented Generation (RAG) applications. Here's what I learned: Key Evaluation Metrics: ✅ Groundedness: Ensures responses are factually supported by context ✅ Answer Relevance: Measures if responses directly address queries ✅ Context Relevance: Evaluates quality of retrieved information The Process: Set up automated evaluation pipelines using TruLens Track performance across different configurations Use metrics to optimize: Chunking strategies Retrieval parameters Context window sizes Embedding models Benefits: Data-driven optimization Automated quality assessment Systematic performance tracking Clear metrics for improvement Result: More accurate, relevant, and reliable RAG applications with measurable improvements in response quality. #AI #MachineLearning #LLM #RAG #DataScience #TruLens https://lnkd.in/gHQnjREi

3

🚀 Built a Transformer Model from Scratch! 🚀 I recently delved into constructing a Transformer Model from the ground up, embracing the transformative power of natural language processing. Here are 8 key takeaways from this enlightening journey: 🔹 Tokenization Matters: Effective tokenization is vital for meaningful sequence modeling, exploring methods like WordLevel and Whitespace. 🔹 Greedy Decoding: Implemented a step-by-step greedy decode algorithm for precise sequence generation. 🔹 Masking for Causality: Utilized causal masks to maintain the autoregressive nature, focusing on past tokens. 🔹 Cross-Attention Mechanism: Explored the intricate connection between source and target sequences through cross-attention mechanisms. 🔹 Handling Variable Sentence Lengths: Dynamically managed varying sentence lengths by determining optimal source/target lengths. 🔹 Loss Function Design: Enhanced training stability and generalization with label smoothing and padding token exclusion. 🔹 Dynamic Learning Rates: Employed learning rate scheduling to fine-tune optimization and prevent overshooting. 🔹 Model Checkpoints: Implemented model checkpoints to preserve training progress and prevent data loss. Witness the remarkable progress of my model, elevating from initial predictions to surprising translations in just one epoch of training. Explore the project here: [GitHub - Transformer-based-Translation-Model](https://lnkd.in/dQd5ecwC) If anyone wants to pair on exciting ML projects feel free to connect and message me 😊!

35

6 Comments