Abhinav Kimothi

Impact of Format Restrictions on Large Language Model Performance 💥💥 GET FULL SOURCE CODE AT THIS LINK 👇👇 👉 https://lnkd.in/de-F9cZC Format restrictions play a significant role in shaping the performance of large language models. These models, designed to understand and generate human-like text, have limitations imposed by their input and output formats. In this description, we'll delve into the impact of various format restrictions on LLM (Large Language Model) performance. Firstly, characters per prompt limit affects the complexity and length of tasks. Shorter prompts with well-defined formats lead to improved model performance. (1) To enhance your learning, consider working on related tasks with clear objectives and succinct instructions. Moreover, token length limitations can significantly impact generating long and complex responses. The use of word embeddings and subword representations help mitigate this effect, but it is essential to understand the tradeoffs. (2) For effective practice, concentrate on understanding tokenization and embedding techniques. Lastly, heavy dependency on transfer learning poses another challenge. Limited availability of high-quality, appropriately formatted labeled data hampers model performance. (3) Curate a dataset and format it correctly for maximum benefits, aiming to fill gaps in the current data sources. [1] Grave, D., et al. "Vulture Transformers: The Merits and Limits of Long-Form Text Processing." arXiv preprint arXiv:2202.07751 (2022). [2] Devlin, Jacob, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of the 2019 Conference on Machine Learning, PMLR, 2019. [3] Merity, Simon, et al. "How to Fine-Tune BERT for Natural Language Understanding." arXiv preprint arXiv:1908.09463 (2019). Additional Resources: - [Surface Forms: A New Class of Synthetic Training Data for Transfer Learning 🤗 Transformers](https://lnkd.in/dSubpC4v) - [Understanding the token, segment and character IDs](https://lnkd.in/di7Nau-j #STEM #Programming #Technology #Tutorial #impact #format #restrictions #large #language #model #performance Find this and all other slideshows for free on our website: https://lnkd.in/de-F9cZC #STEM #Programming #Technology #Tutorial #impact #format #restrictions #large #language #model #performance https://lnkd.in/dVw7vsPW

1

🧠 Apple Intelligence is a GPT wrapper.  💤 GPT : We will not tell what Data we used Bias: Who cares 🤷🏻‍♀️ 🤷🏻‍♂️ GUP Designers / Manufacturers : we are building better G/.. Processing Unit (PU) Data Privacy and Security will be a big question with the Apple and Open AI Engagement. No one has questioned OpenAI , Anthropic ... on biases ❓ All LLMs are biased towards the origin of the open-source data, the mindset of the authors, and people who can write/speak English and had early access to the internet. All LLMs exhibit biases stemming from the data they were trained on, which is often skewed towards Western, English-speaking sources and perspectives. This can lead to a lack of knowledge or misrepresentations about other regions, cultures, and languages, particularly those from the Global South like India and Africa. The biases in LLMs are a significant issue that needs to be addressed through more diverse and representative training data, as well as techniques like prompt engineering and human oversight. Relying solely on existing models without accounting for their limitations and biases can perpetuate harmful stereotypes and marginalize underrepresented groups. Hope Apple and Open AI collaboration bring some solutions towards. Sources for further digging: https://lnkd.in/gDAJkgei https://lnkd.in/gPYStYEi https://lnkd.in/gB6H3-wj

6

Here is a wonderful paper Published last week by MIT, Drexel, Harvard, BU & Lehigh on LLM benchmarks using Game Theory with strategic scenarios & Possibility of Nash equilibrium with LLM reasoning. :) https://lnkd.in/dGKSC6uK #ai #data #datascience #ml #gametheory This is probably the start of multiagent systems/LLMs coming together at the same time for a given use case :)

20

📚 Book #10 | Turning #Data into real #Money This is best book on Data I have read in years, BUT … .. it was published just 3 years too late. But let me come to that point later. 💦 You can go to book store and discover hundreds books on data best practices and advice how to undergo data transformation. Most of them, are just by authors with narrow or isolated experience in how to make real splash with data. 💵 Collective of Barb Wixom ,Cynthia Beath and Leslie Owens authors did something completely different. Coming from scientific background their turned curious question “Ok, but how does data make REAL (monetary) difference?” into solid research question. They studied large set of companies trying (and failing) to reap monetary benefits out of data initiative. And benchmarked what works. 👩‍🍳 As a person who led data transformation in 8 different industries, I had my portion of trying to take data airplanes off the grounds. I Often can debunk out-of-bat “silver bullet advices” from popular data literature as either shallow or down right charlatan. But DATA IS EVERYBODY’s BUSINESS book is really different. It´s great cookbook on how to build data programs that deliver. Data programs that can bootstrap itself, that don’t need million dollar budget before giving some money back to company. 🔩 Book is well structured into giving first understanding of #WHAT are necessary skills/changes to make. Then taking you through #HOW #TO #WRAP it (or weave into) your company. It also gives full scope of options to #COLLECT income from data, many of you probably haven’t thought of yet. It also adds strategies to put the #right (#purple) #PEOPLE behind the steering wheels. 💯 In my past career I cumulatively implemented probably 85% of suggestions given in this book: from data solutions, to tweaking customer behavior through data to selling data sets. And I did not find single case, where what these ladies suggest, would divert from my own experience on matter. This is literally the best book on data strategy I read in several years (I read about 30/y of them). BUT I CAN’t GIVE IT 10/10, not even 9/10. You wonder why? 🏛️ This volume covers perfectly the data strategy in BEFORE AI era. I am not sure if the research took longer or just the idea came too late, but both the language used (heavy on data analyst and generic science) as well as examples used are for end-of-BI-looking-into-some-advanced-analytics stage of company. 🩼 That having said, if your company struggles with data basics, this is still fantastically useful book and you should definitely read it. May you need to compete with your market rivals on AI features, you might still benefit from the early chapters, but soon you realize examples and auggestions of this book might not cut it for your organization. When preparing our new AI&Data strategy in CommentSold we still got inspired, but you conciously need to make “and what about AI version of this” mental extension.

6

Got a question, Can SECTOR evaluate summarization use cases? Yes, as SECTOR breaks down summary and looks at relevant part of context to compare This Engineering + unique statistical metrics makes it suitable for all use cases. All of this without having a secondary LLM for evaluation Here is the example to evaluate summary - https://lnkd.in/gU-9hqf7 See a sample output below #generativeai ##artificialintelligence

11

6 Comments

I wrote a piece for the Economic Times' ET Edge Insights discussing why regulation and responsible AI is priority as AI diffuses further across industry. This is very relevant in the context of developing AI regulation. As the EU AI Act becomes embedded and the passages of the similar AI regulatory acts continues their journeys through the US, UK and other parliaments, this will become more relevant for every business utilising AI. https://lnkd.in/dHA_7fun #ResponsibleAI #AIEthics #AIRegulation

37

5 Comments

📢 Best Advice from a Senior Data Scientist Packed in One Podcast: Insights from Abhishek Das!...🌟 🔷 Speaker: Abhishek Das Senior Data Scientist at Aptus Data Labs Ex-HSBC 9+ Years of Experience in Advanced Analytics and Machine Learning 🔷 Hosted by: Avinash Dara Here are some highlights from his talk: 🔷 His Personal Life: • He came from an Electronics and Communication Engineering background with an interest in research. • He was always interested in the business field to scale. • In his college placements, he joined e2open where the environment was new to him. He took some time to understand and scale in that industry. After that, his data science career started. In addition to this, he also started writing a book. 🔷 He worked across diverse industries such as: • Transportation, pharmaceuticals, manufacturing, supply chain, e-commerce, and banking. • Each sector has shown him the immense value data science brings to businesses, driving growth through informed decision-making. 🔷 He explained what Data Science is: • Data science is leveraging data to add value to a business. • He gave a practical example like Uber and Ola, which calculate the ride price based on other factors. 🔷 Career Paths in Data Science: 1. Data Engineer o Roles: Building and maintaining data infrastructure o Skills to acquire: Data engineering tools and technologies 2. Data Analyst o Roles: Extracting data, proficient in SQL, visualization, and providing insights o Skills to acquire: Strong SQL skills, data extraction abilities, and expertise in visualization tools 3. Data Scientist o Roles: Focusing on machine learning, algorithms, optimization, and prediction o Skills to acquire: Proficiency in machine learning, algorithms, and predictive analytics 4. ML Engineer o Roles: Researching, creating, and improving algorithms, and deploying models o Skills to acquire: Deep understanding of algorithm development and deployment 🔷 His advice for interview tips: 1. Be truthful about your skills. 2. Clearly indicate your proficiency levels. 3. Showcase 2-3 relevant projects, including ML projects. 4. Align your projects with industry-relevant problems, such as financial predictions or sales forecasting. 🔷 His approach for finding opportunities: • Apply to startups and MNCs. • Optimize your resume on LinkedIn. • Network with hiring managers and senior employees for referrals. • Target analytics-driven companies. 🔷 He also explained about LinkedIn: • LinkedIn is an incredible platform for networking and professional development. • Maintain a problem-solving mindset and continually showcase your skills and projects. #datascience #dataengineering #mlengineer #machinelearning #ccbp #advise #podcaste #datascientist #learning

35

Something I've written on how a T20 batting innings can be assessed. This framework can be extended to assessment of individual players as well, which I will do in the susequent posts. For now, please feel free to comment and if you like the content, please subscribe. #cricket #cricketanalytics #data #dashboards #datavisualization #sportsanalytics https://lnkd.in/gb7T-UzJ

28

6 Comments

Unraveling Deep Learning: A Deep Dive into Computational Graphs and Automatic Differentiation by Ritika Kumari Post: In her recent article, Ritika Kumari does an exceptional job of breaking down the intricate workings of deep learning models, particularly focusing on how computational graphs and automatic differentiation streamline the process. Here’s why you should give it a read: 1. Clear Explanation of Deep Learning Models: Ritika starts by summarizing the structure of a typical deep learning model, making it accessible even to those new to the field. She provides a concise overview of how neurons in hidden layers compute outputs, setting the stage for more complex concepts. 2. Insightful Discussion on Computational Graphs: Ritika dives into computational graphs, explaining how breaking down the network into basic operations and representing them as a Directed Acyclic Graph (DAG) enhances efficiency. She highlights how storing intermediate gradients speeds up backpropagation, which is crucial for training large neural networks. 3. Comparison of Differentiation Methods: The article compares numerical, symbolic, and automatic differentiation methods, offering readers a clear understanding of their pros and cons. Ritika writes, “This is where automatic differentiation (AD) comes into play. AD combines the precision of symbolic differentiation with the computational efficiency of numerical differentiation.” 4. Introduction to Jacobians: Ritika skillfully introduces the concept of the Jacobian matrix, explaining its importance in managing the complexity of vectorized operations in neural networks. Her explanation makes this advanced topic much more digestible. 5. Practical and Visual Approach: The use of figures and diagrams, as seen in her explanation of automatic differentiation, adds a visual clarity that complements the textual content, making complex ideas easier to grasp. Conclusion: Ritika Kumari’s article is a must-read for anyone interested in the inner workings of deep learning frameworks like TensorFlow and PyTorch. It not only explains the concepts but also provides practical insights into their applications. #DeepLearning #MachineLearning #ArtificialIntelligence #ComputationalGraphs #AutomaticDifferentiation #ai #ml

8

Enhancing the accuracy of Large Language Models (LLMs) is a key focus in ongoing AI research. Previously, we have used a Critic model along with Reward Engineering to identify incorrect responses and incrementally fine-tune the LLM. The introduction of Thought Preference Optimization (TPO) advances this approach by having the LLM generate preliminary thoughts which a judging model then refines into better responses. This method is akin to the human practice of "thinking before responding," a critical safeguard in high-stakes situations, pushing LLMs to adopt a similar evaluative process. Training an LLM to pause, generate internal thoughts, and evaluate these before delivering a final response is complex and resource-intensive. It typically requires large datasets of human-annotated thoughts, which are scarce across diverse domains. Chain of Thought (CoT) prompting is another method where the model is instructed to articulate intermediate reasoning steps, structuring its approach to tasks. While CoT has proven effective in fields like mathematics and logic that require clear reasoning steps, it often underperforms in more subjective areas such as marketing or creative writing. This limitation is exacerbated by training datasets that generally contain human responses without the underlying thought processes, restricting the model’s ability to refine its reasoning across varied domains. Thought Preference Optimization (TPO) aims to equip LLMs with the capability to generate and refine internal thoughts before producing a response. Crucially, it operates independently of human-annotated datasets. It begins by instructing the model to divide its output into two distinct parts: the thought process and the final response. The model generates multiple thoughts for each instruction, which are then evaluated through performance optimization to select the best thought-response pair for further training iterations, allowing the model to gradually enhance its reasoning capabilities. TPO employs a reinforcement learning technique, enabling the model to learn from its thought process. The model generates preliminary thoughts before responding, and a judging model evaluates the resulting responses. This iterative refinement process leads to more thoughtful and accurate answers. The effectiveness of TPO was tested on two benchmarks: AlpacaEval and Arena-Hard. On AlpacaEval, TPO achieved a win rate of 52.5%, outperforming the direct response baseline by 4.1%. Similarly, it registered a win rate of 37.3% on Arena-Hard, surpassing traditional methods by 4.3%. One of the most significant findings is that thinking-based models generally outperformed direct response models across various domains. Even in non-reasoning tasks like creative writing, TPO-enabled models could plan their responses more effectively, leading to better outcomes. https://lnkd.in/d5kVzjxv #llm #ai

3

1 Comment

Sharing my thoughts on random vs. time-series splits in machine learning problems with a time-series component. This topic sometimes appears in my feed, and some shared recommendations can be misleading for new data scientists who may not yet understand the reasoning behind them. My TL;DR: Don’t overcomplicate things; stick to the basics. If your problem has a time component, treat it as a time-series problem and perform a time-series split. PS: I didn't want to go into detail but normally we want to discard recent data for fraud detection because chargebacks can take up to 3 months. This also happens for other machine learning problems. May talk about this in the future .

36

4 Comments

Knowledge Graphs (KGs) have long been the unsung heroes behind technologies like search engines and recommendation systems. They store structured relationships between entities, helping us connect the dots in vast amounts of data. But with the rise of LLMs, KGs are evolving from static repositories into dynamic engines that enhance reasoning and contextual understanding. This transformation is gaining significant traction in the research community. Many studies are exploring how integrating KGs with LLMs can unlock new possibilities that neither could achieve alone. Here are a couple of notable examples: • 𝐏𝐞𝐫𝐬𝐨𝐧𝐚𝐥𝐢𝐳𝐞𝐝 𝐑𝐞𝐜𝐨𝐦𝐦𝐞𝐧𝐝𝐚𝐭𝐢𝐨𝐧𝐬 𝐰𝐢𝐭𝐡 𝐃𝐞𝐞𝐩𝐞𝐫 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Researchers introduced a framework called 𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐆𝐫𝐚𝐩𝐡 𝐄𝐧𝐡𝐚𝐧𝐜𝐞𝐝 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐀𝐠𝐞𝐧𝐭 (𝐊𝐆𝐋𝐀). By integrating knowledge graphs into language agents, KGLA significantly improved the relevance of recommendations. It does this by understanding the relationships between different entities in the knowledge graph, which allows it to capture subtle user preferences that traditional models might miss. For example, if a user has shown interest in Italian cooking recipes, the KGLA can navigate the knowledge graph to find connections between Italian cuisine, regional ingredients, famous chefs, and cooking techniques. It then uses this information to recommend content that aligns closely with the user’s deeper interests, such as recipes from a specific region in Italy or cooking classes by renowned Italian chefs. This leads to more personalized and meaningful suggestions, enhancing user engagement and satisfaction. (See here: https://lnkd.in/e96EtwKA) • 𝐑𝐞𝐚𝐥-𝐓𝐢𝐦𝐞 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠: Another study introduced the 𝐊𝐆-𝐈𝐂𝐋 𝐦𝐨𝐝𝐞𝐥, which enhances real-time reasoning in language models by leveraging knowledge graphs. The model creates “prompt graphs” centered around user queries, providing context by mapping relationships between entities related to the query. Imagine a customer support scenario where a user asks about “troubleshooting connectivity issues on my device.” The KG-ICL model uses the knowledge graph to understand that “connectivity issues” could involve Wi-Fi, Bluetooth, or cellular data, and “device” could refer to various models of phones or tablets. By accessing related information in the knowledge graph, the model can ask clarifying questions or provide precise solutions tailored to the specific device and issue. This results in more accurate and relevant responses in real time, improving the customer experience. (See here: https://lnkd.in/ethKNm92) By combining structured knowledge with advanced language understanding, we’re moving toward AI systems that can reason in a more sophesticated way and handle complex, dynamic tasks across various domains. How do you think the combination of KGs and LLMs is going to influence your business?

432

12 Comments

F1-Score vs Accuracy: Why F1-Score Should Be Your Focus. In machine learning, it's tempting to focus on ACCURACY—the percentage of correctly predicted instances. But what if I told you that accuracy isn't always the best indicator of your model's performance? In cases where the data is imbalanced (e.g., fraud detection, disease diagnosis), accuracy can be misleading. Imagine a scenario where only 5% of transactions are fraudulent—if your model predicts "non-fraudulent" 100% of the time, you’ll get 95% accuracy but miss all the critical fraud cases! When it comes to the F1-score: - Precision tells you how many of your positive predictions were correct. - Recall tells you how many of the actual positives you correctly identified. The F1-score balances both precision and recall, giving you a better sense of how well your model performs—especially when false positives and false negatives matter. If you're working on models where missing a positive instance could have serious consequences (like healthcare, security, or finance), the F1-score should be your go-to metric. Focus on the right metrics to make smarter models and better decisions. #MachineLearning #DataScience #AI #F1Score #ModelEvaluation #TechInsights #DataDriven #BrainK

10

Excited to share our latest paper titled - 𝐇𝐨𝐰 𝐭𝐨 𝐂𝐡𝐨𝐨𝐬𝐞 𝐚 𝐓𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝 𝐟𝐨𝐫 𝐚𝐧 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐌𝐞𝐭𝐫𝐢𝐜 𝐟𝐨𝐫 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬. This work addresses the critical gap in research on setting robust thresholds for LLM evaluation metrics, which are essential for the reliable deployment of LLMs. Drawing inspiration from model risk management practices in regulated industries like finance, we propose a systematic methodology to define these thresholds. The approach involves identifying risks associated with the specific application, understanding stakeholders' risk tolerance, and applying statistically rigorous procedures using ground-truth data.

58

4 Comments

Much needed pointers for enterprises considering operationalization of LLMs.

7

Sharing update from SSRN publishing... "Dear Ashutosh Bhatnagar: Your paper, "NEIGHBORHOODS OF PRIMES", was recently listed on SSRN's Top Ten download list for: Discrete Mathematics & Combinatorics eJournal and Number Theory eJournal. As of 02 October 2024, your paper has been downloaded 16 times. You may view the abstract and download statistics at: https://lnkd.in/eMmhurr4. Top Ten Lists are updated on a daily basis. Click the following link(s) to view the Top Ten list for: Discrete Mathematics & Combinatorics eJournal Top Ten and Number Theory eJournal Top Ten. Click the following link(s) to view all the papers in: Discrete Mathematics & Combinatorics eJournal All Papers and Number Theory eJournal All Papers. To view SSRN's Top Ten lists for any network, subnetwork, eJournal or topic on the Browse list (reachable through the following link: https://lnkd.in/eEmS8twm), click the "i" button to the right of the name, and then select the "Top Downloaded" link in the popup window. Your paper may be included in future Top Ten lists for other networks or eJournals. If so, you will receive additional notices at that time. Questions? Visit the SSRN Support Center. This email helps you gauge the impact of your paper. To unsubscribe from ALL notifications, including SSRN eJournals, click here. Thank you, The SSRN Team"

10

3 Comments