Christina Boididou

Lots of genAI makes me eye roll so hard I risk tearing my optic nerve. ML has its place, GenML only looks like it has a place. On the one hand I should keep quiet and land a fat pay day. On the other I have a sense of ethics and want the people paying me to benefit from the work I do. There is a reason the number one item on our ML project check list is a cost benefit analysis.

4

Last month, DeepMind released FACTS Grounding - a benchmark for evaluating Factuality and Quality of LLM outputs...and they avoided most of the common benchmark pitfalls that I check for! (https://lnkd.in/eT88wNHj) - ✅ *Reproducibility*: Official evaluation code is available (https://lnkd.in/exp3kP2c) - ✅ *Test Data Leakage*: About half of the evaluation examples are private - they can not accidentally be scraped or included in an instruction tuning dataset (unlike GSM8K - https://lnkd.in/ec_a87ZJ) - ✅ *Construct Validity*: The benchmark design involves a step to check if the output is "eligible" (i.e. that it contains the relevant information) - this discredits the responses that are "factual" but contain no useful information. The developers, also, did not include examples that involved creativity, mathematics, or complex reasoning to target factuality exclusively. - ✅ *Saturation*: Over time SotA Models hit a ceiling on performance, and no longer represent a useful proxy for performance. The top models currently score in the low 80s on this benchmark, which shows that there is still room for improvement. - ✅ *Process and Limitation Documentation*: The benchmark developers clearly document the process for assembling the benchmark and explain how they handled limitations. For example: Evaluation is done using LLM-as-a-Judge, and "judges" are known to favor outputs from the same model. The evaluation uses an ensemble of 3 different models to reduce this bias. - 🚫 *Variance*: Each input prompt is only run once for every model. Given that LLMs produce different outputs over multiple runs of the same input, I would have liked to see that element incorporated in evaluation. However, given that LLM-as-a-Judge methods are used during grading, the one-run outputs may be fairly reliable. I am still finalizing my checklist of common benchmark pitfalls, so stay tuned.

18

SparseEmbed. This work combines the strengths of sparse (SPLADE) and dense (ColBERT) representations for first-stage retrieval. TL;DR: - For a given sentence, sparse vectors are generated (similar to SPLADE) - For the same sentence, contextualized embeddings are generated for each token (similar to ColBERT). However, the attention layer only uses the logits of the sparse vectors (including token expansions) to generate the embeddings. - This results in faster training and smaller contextualized embeddings for each token. - Retrieval is hybrid. First, keyword matching matches the query to document expanded tokens. Then, scoring is computed based on the dot products between query and document contextual embeddings.

47

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

🚀 𝗚𝗼𝗼𝗴𝗹𝗲 𝗗𝗲𝗲𝗽𝗠𝗶𝗻𝗱'𝘀 𝗔𝗜 𝗪𝗲𝗮𝘁𝗵𝗲𝗿 𝗙𝗼𝗿𝗲𝗰𝗮𝘀𝘁𝗶𝗻𝗴 𝗧𝗲𝗰𝗵 𝗪𝗶𝗻𝘀 𝗧𝗼𝗽 𝗨𝗞 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗔𝘄𝗮𝗿𝗱 Google DeepMind’s AI weather forecasting system, GraphCast, has won the prestigious 2024 MacRobert Award, marking a significant breakthrough in weather prediction. 🔑 Key Highlights: 1️⃣ 𝗥𝗲𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝗮𝗿𝘆 𝗧𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝘆: GraphCast leverages advanced machine learning and vast datasets to deliver highly accurate and timely weather forecasts. 2️⃣ 𝗟𝗶𝗳𝗲-𝗦𝗮𝘃𝗶𝗻𝗴 𝗜𝗺𝗽𝗮𝗰𝘁: The system enhances the ability to predict severe weather events, enabling early safety warnings and better resource allocation, potentially saving lives. 3️⃣ 𝗔𝘄𝗮𝗿𝗱 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝗼𝗻: The MacRobert Award, presented by the Royal Academy of Engineering, honors outstanding engineering ingenuity, commercial success, and social benefit. 4️⃣ 𝗘𝘅𝗽𝗲𝗿𝘁 𝗘𝗻𝗱𝗼𝗿𝘀𝗲𝗺𝗲𝗻𝘁: GraphCast is praised for significantly improving the speed and precision of weather forecasting, crucial in the era of climate change. GraphCast showcases how engineering innovation can drive economic growth and foster a sustainable future. Precise weather forecasting is more critical than ever in addressing ongoing environmental challenges. 🌍🔬 #Innovation #Forecasting #AI #ArtificialIntelligence

3

Spent some time today experimenting with GraphRAG - https://lnkd.in/eVqcVr3A It's a really interesting idea. Essentially, by employing a better data structure (like a graph in this case), one can improve the model's semantic understanding of the data. This is because now it has easy access to abstractions it didn't before. I indexed my favourite plasma physics paper that I read thousands of times during grad school and built simple RAGs - one the standard way, and one using GraphRAG and the difference is tremendous! To the question "what are the main themes of this article?", baseline RAG picked out different phrases and gave an answer that on the surface sounds reasonable but isn't quite there. It also mentioned one theme where the model just gets it completely wrong. Whereas, graph RAG did manage to extract a few relevant themes. Caveat: I hacked this together very quickly and haven't spent much time improving the data extraction. I reckon the graph RAG performance will improve a lot with better data. Planning to try and find some time in the coming days to (1) improve the data quality (2) build a nicer UX to easily compare the two applications The code and screenshots of results are on github under MIT license, as always: https://lnkd.in/eVkvJrcM

34

3 Comments

🤖 A Complete Guide to BERT and Fine-tuning! 🧠 I've finally finished Part 4 of my LLMs from Scratch series - a collection of articles where I take advancements in Natural Language Processing and break them down with diagrams and code. This article focuses on Google's BERT model, including: 📚 A Complete History of BERT - what is it and why is it important? 🤖 Architecture Explanation 💥 Detailed Fine-tuning Guide This article breaks down the NLP landscape at the time of BERT's creation, and use this to explain why BERT has the architecture it does. We then take a deep dive into the inner workings of BERT, including its special tokens such as [CLS] and [SEP], then finish by writing a fine-tuning pipeline which can be used to fine-tune BERT on any downstream task. If that sounds interesting to you, take a look at my Medium post below! https://lnkd.in/diq3B8NN

10

This is really awesome. I used notebookLM to summarise a recent paper that shows how o1 like model can be reproduced. After providing the pdf from Arxiv, I noticed that the generated podcast was almost 49 minutes long. Very different from the earlier 10-12 minute version. And another impressive feature was the interactive mode where you can interject the two hosts and ask a question which they very graciously respond to. I wanted to share the audio summary but YT didn’t allow me to upload the WAV file. YT expects a video mp4 file. So I found free videos from Pexels and stitched them together with ffmpeg. The videos are meditative acquarium shots of fish swimming around. And with the audio discussing reproducing o1 as an engaging podcast episode, the video keeps the distraction to the minimum. The paper itself is a great read but if you have a 30-40 minute commute, or want to watch some cool fish swimming around and learn about o1, check out

12

I'll never be able to read the word "delve" again without immediately thinking: Ah, this is AI writing. Anyway! A new paper has been published, along with fine-tuned models for generating really long outputs. The researchers weren't satisfied with the fact that current LLMs could technically produce much more output than they normally do. ChatGPT and similar models typically generate around 2000 words, despite being capable of producing far more tokens. Their hypothesis: The models don't encounter longer outputs during training. Initially, the researchers used prompt chaining to make ChatGPT-4o generate texts longer than usual. After succeeding, they created a training dataset with longer texts and fine-tuned two current models, Llama 3 and GLM 4. I tried the GLM 4 on Hugging Face and fed it a brief book summary of Jonathan Safran Foer's "Eating Animals." The result is ten pages of text, approximately 30,000 characters. The, well, good news: The result isn't a copy of Foer. It's AI slop, barely readable at best. Which is also the bad news, I guess.

30

7 Comments

🔍 Google's Gemma Scope: Illuminating the Inner Workings of Large Language Models 🧠 Google researchers have developed a powerful new tool called Gemma Scope, designed to shed light on how each layer in Gemma 2 large language models responds to input tokens. 📊 Key insights: - Sparse autoencoders (SAEs) can transform embeddings into interpretable representations - Each index in the transformed embedding corresponds to a distinct concept - SAE weights indicate the strength of each concept in the input Gemma Scope enables: - Manual and automatic labeling of concepts in each layer - Steering the model by adjusting SAE outputs to generate concept-specific text This groundbreaking tool paves the way for answering critical questions about LLMs, such as how fine-tuning and chain-of-thought prompting influence a model's internal representations. 🤔 #GemmaScope #LargeLanugageModels #TransformerInterpretability #GoogleAI #MachineLearningResearch

1

Many years ago, when I first encountered Data science, especially on its claims on possible sentiency, I thought it was all stupid and I even wrote LinkedIn articles as such (since then deleted those articles to avoid confusion!). Ignoring data science, I learnt Cloud technologies, Kubernetes, DevOps, Blockchain etc. Subsequently, when I happened to study data science from job search perspective, and when I started studying Deep Learning, I realized the huge potential of AI, and wrote an article avowed by its future possibilities. My intuition was proved when ChatGPT arrived. Though ChatGPT was far beyond the expectations, it was not totally surprising. However, owing to its phenomenal capabilities, the doubt remained as to whether an 'artificial consciousness' has been created - implying a greatest technological feat of humanity, or was it just a pseudo-science, involving external consciousness, such that in its absence, the AI programs will become useless, chaotic and at least not as good as they are now. I would like to continue to believe that it is not external consciousness but the concept of emergent abilities are indeed true (refer to my articles on this topic), and that we are indeed actually making digital brains. When I listen to AI leaders talking about its potential, I think they are talking from the perspective of creation of digital brain (that can even somehow harness cosmic consciousness 😀 ) and when I write here about AI, I write it from this perspective only! However, if we assume that the second premise is true - that only external consciousness is involved, there is another interesting thing I thought of writing. Long time ago, I used to frequently receive emails regarding "Leadership program for Ethical AI" from a reputed institution. I used to wonder why am I receiving promotion for this program as against any other AI program! Now, I guess there could even be a misunderstanding with regard to 'what really is ethics in AI' that many talk about! 😀 Assuming that AI is not any great technological feat of humanity and that it functions solely, by capturing / deploying external consciousness that could even exhibit 'power of consciousness', the question of ethics can have different connotation. If we go by the definition of consciousness that says emotions and feelings are integral to consciousness, and also by the tenet that consciousness cannot exist independent of a soul, the question arises how ethical is it to deploy external soul consciousness into machines? Are we torturing innocent consciousness / souls by deploying them into machine programs for our selfish material benefits? Is it sustainable at all in the long term? May be when we talk about Ethics in AI, we should be talking about this! Not about using AI ethically - that is completely under our control. I continue to hope that we have indeed achieved a greatest scientific milestone, and hence there is no sufferings involved in this AI phenomenon.

4

2 Comments

Excited to share and interact with fellow Data Scientists at the Data Science Festival in London on the 18th May! https://lnkd.in/e6HmhQCM Luca Valer and I will be talking about the use of LLMs for Topic Modelling by considering domain specific semantic textual similarity (STS) and the fine tuning of open source LLMs such as Llama2 for multi-class text classification purposes where the context length is longer than that of BERT (512 tokens). #DigitalMarketing #DataScience #GenerativeAI #AI #Informa

18

My 2nd blog post for the Vizuara Substack is live now This blog has been written in a lot of detail about the Embeddings in NLP: Firstly, we went through the evolution of approaches to work with texts. - Bag of words - tf-idf -Word2Vec -Transformers and sentence Embeddings. Then, we discussed how to understand whether texts have similar meanings to each other. After that, we saw different approaches to text embedding visualisation. Here is the link to the blog post: https://lnkd.in/gn3YGZqM Raj Abhijit Dandekar #nlp #embeddings #vectors #word2vec #tf-idf #bagofwords #sentenceembeddings

32

1 Comment

I’ve never thought of Data Science as just training ML models - to me it’s always been so much more. For me, a broad description would be: “Using data to help businesses make better strategic decisions, to understand their customers, and to predict what is likely to happen next” There is so much value in this, and it will always be in high demand - regardless of what tools help you do it.

90

8 Comments

𝗙𝗿𝗼𝗺 𝗣𝗿𝗶𝘃𝗮𝗰𝘆 𝘁𝗼 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆: 𝗧𝗮𝗰𝗸𝗹𝗶𝗻𝗴 𝗔𝗜 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 𝗮𝘁 𝗡𝗛𝗦 𝗘𝗻𝗴𝗹𝗮𝗻𝗱 I’ve recently written two articles for the NHS England Data Science Website based on some of the really interesting work I’ve been doing! The first: Investigating Annotation Tools for Named Entity Recognition (1). If you like Generative AI, entity extraction and identifying privacy risks - this is for you. The second: The RISE Tool (2) - An Easy Way for Testers and Assurers to Evaluate AI Classifiers. This covers AI evaluation, assurance, image classifiers and explainability. The NHS England Data Science team have a whole host of interesting projects, you can read about them all on our website (3). 1 - https://lnkd.in/dNaUXsgv 2 - https://lnkd.in/dEjp88ra 3 - https://lnkd.in/dYg53A4j

19