Antreas Oikonomou

Wonderful article here from Dhanush Kishore on the statistical engine we have developed at Skyscanner to evaluate A/B tests and estimate uncertainty:

70

2 Comments

When I began my journey into #MachineLearning about five years ago, my first experiences were with #Pandas and #Scikit-Learn. It all started with simple commands like `pd.read_csv`, `model.fit(X, y)`, and `model.predict(X)`. However, I quickly realized that this surface-level knowledge wasn't enough to ace interviews. Determined to deepen my understanding, I embarked on a self-study journey to master the mathematics behind ML. I faced two significant challenges: math (which I never excelled at) and coding (as I couldn't find a comprehensive, line-by-line implementation for various algorithms). To address these, I started documenting my learnings, which eventually became [mlguidebook.com](https://mlguidebook.com). Check out some of the algorithms I've implemented: 1. [Linear Regression](https://lnkd.in/gwBQ3U8U) 2. [Logistic Regression](https://lnkd.in/gAxX7Jvx) 3. [Logistic Regression Using Neural Network](https://lnkd.in/gjvhSghp) 4. [Neural Network Using #Numpy](https://lnkd.in/g3eNiEex) 5. [Support Vector Machine](https://lnkd.in/gyVYNe8H) 6. [K-Nearest Neighbors](https://lnkd.in/g7ApDxkz) 7. [Decision Tree](https://lnkd.in/gK3SezvX) 8. [Random Forest](https://lnkd.in/grwE_DTJ) 9. [Gradient Boosting](https://lnkd.in/gTTNXjfh) 10. [K-Means](https://lnkd.in/ghtsCnnh) Explore these resources and enhance your understanding of machine learning from the ground up! #DataScience #AI #LearningJourney Rewritten By ChatGPT 🤖

26

3 Comments

My 2nd blog post for the Vizuara Substack is live now This blog has been written in a lot of detail about the Embeddings in NLP: Firstly, we went through the evolution of approaches to work with texts. - Bag of words - tf-idf -Word2Vec -Transformers and sentence Embeddings. Then, we discussed how to understand whether texts have similar meanings to each other. After that, we saw different approaches to text embedding visualisation. Here is the link to the blog post: https://lnkd.in/gn3YGZqM Raj Abhijit Dandekar #nlp #embeddings #vectors #word2vec #tf-idf #bagofwords #sentenceembeddings

32

1 Comment

Recently I've been playing with "Swarm Intelligence", wondering in the back of my mind how I might apply it to my "day job" and merge with modern methods in the age of transformers...etc. Today I came across a paper (https://lnkd.in/g_bUdGB7) in the TL;DR newsletter that uses many inferences from small models to generate much better solutions than individuals, and at 3x lower cost than using a large model. And it is merely generating several samples independently without any interaction among the models. What if we instead applied a swarm intelligence algorithm to this approach so that inference N was informed by all the prior inferences? Would we better navigate the exploit/explore trade-off? This is the sort of place my video series will be heading. Relevant quote from the abstract: "...when it comes to inference, we often limit the amount of compute to only one attempt per problem. Here, we explore inference compute as another axis for scaling by increasing the number of generated samples. Across multiple tasks and models, we observe that coverage - the fraction of problems solved by any attempt - scales with the number of samples over four orders of magnitude. In domains like coding and formal proofs, where all answers can be automatically verified, these increases in coverage directly translate into improved performance. When we apply repeated sampling to SWE-bench Lite, the fraction of issues solved with DeepSeek-V2-Coder-Instruct increases from 15.9% with one sample to 56% with 250 samples, outperforming the single-attempt state-of-the-art of 43% which uses more capable frontier models."

50

3 Comments

Just Attended a talk by Gabriele Biagini arranged by Amazon Web Services (AWS) where he shared insights on how to build production ready GenAI applications. He explained various design choices in the architecture. Here were the key highlights: Key highlights: 🌐 Real-time Communication: WebSocket APIs enable two-way, real-time communication between clients and servers. 💻 Scalable Middleware: Powered by AWS Lambda, Amazon EC2, and AWS Fargate for a serverless, scalable backend. 📊 Connection Management: DynamoDB manages WebSocket connections, while Secrets Manager handles API key security. 🤖 AI Integration: Seamless integration with LLMs like OpenAI, Hugging Face, Amazon Bedrock, and SageMaker for real-time, AI-driven responses. 💬 Real-Time Use Cases: Ideal for chatbots, real-time customer support, and dynamic apps needing continuous, instant feedback. #AWS #WebSockets #AI #Serverless #RealTime #LLM #TechInnovation #CloudComputing #Architecture

25

2 Comments

Sharing my latest article on the RAFT: Teach LLMs to be better at RAG Discover how RAFT bridges the gap between fine-tuning and RAG, enhancing model performance by integrating domain knowledge with powerful retrieval techniques. 🔍 Dive into: • The mechanics of RAFT • Benefits over traditional methods • Real-world applications using Meta Llama 2 Read the full article here: https://lnkd.in/g7ZQNE5s #AI #MachineLearning #DataScience #NaturalLanguageProcessing #AIInnovation #MetaLlama2 #RAFT #llm #RAG #genAI #LargeLanguageModels

9

Following on from last week's article, this week I cover Activation Functions

1

Thanks for running the 4 hours: Full-Stack Machine Learning with Metaflow course Hugo Bowne-Anderson and Eddie Mattia. It reconfirmed that #Metaflow from Outerbounds is a great choice to run ML flows here at Awaze. PS: Great code snippets can be found here: https://lnkd.in/e2u-pGx7 #MLOPs #Outerbounds #Metaflow

14

Were you unable to attend this year’s 🔥 ICLR 2024 🔥 at the beginning of May? As part of our work at Echo Analytics , we went on a trip to get you some surprising findings or trends you might have missed ⬇ ✨ On Large Language Models✨ ➡Using LLMs to check and correct their own outputs works well to improve their accuracy: https://lnkd.in/ebGGPtjz https://lnkd.in/e4SveSxA https://lnkd.in/ehYagWc8 https://lnkd.in/ekiNkDYa ➡A multi-agent role-based collaborative framework for LLMs: https://lnkd.in/eEWXWNPc ➡In context Learning in LLMs (most probably) guesses the rules of the task on the fly, and doesn't just recognize the task from a set of previously seen tasks: https://lnkd.in/eRH_vHnK ➡Contrastive instruction tuning, where we maximize (“probability of the right response” - “probability of a response to the opposite instruction”) outperforms vanilla instruction tuning: https://lnkd.in/ekByaUrH ➡Generating task demonstrations with LLMs for LLM few shot prompting is (almost) as good as human-written demonstrations: https://lnkd.in/edaxiEdY ✨ On Time series modeling✨ ➡Time series modeling (even with transformers) does improve if the input series is decomposed: https://lnkd.in/eYBdh8wm https://lnkd.in/eE9Rp7fV https://lnkd.in/eRQbJvGU ➡Foundation Models for time-series are (surprisingly) a valid concept: https://lnkd.in/em49ZsDF https://lnkd.in/ex3VWdAc ✨ Miscellaneous highlights ✨ ➡ Subsampling Data from a data set (e.g. 60%) by only keeping examples that are harder (not hardest) according to a silver labeler can significantly improve results: https://lnkd.in/eRCKB6T6 ➡Training S4 models from scratch leads to better results for S4 than Transformers. BUT, adding Self-Supervised Denoising training for both evens out the results, even when it’s only done on the (small) dataset of the target task: https://lnkd.in/eH6d5xnu It’s also worth mentioning that Kingma and Welling’s work “Auto-Encoding Variational Bayes” recieved the “Test of Time” Award, a well deserved consecration for Variational Autencoders over the last decade 🎊

61

1 Comment

This is a major development for the open-source AI ecosystem. Apple has gone fully open-source! They have just released DCLM, a 7B open-source LLM, including weights, training code, and dataset. They plan to open-source all their models for deployment across their billion devices. Apple's embrace of open-source marks a significant shift. Here’s a summary: → Apple has released and open-sourced their DCLM 7B model, outperforming Mistral/Qwen2/Gemma on various benchmarks. → It is primarily trained on English data with a 2048 context window, and licensed under the Apple Sample Code License. → The release includes an exceptional explanation of the data curation process, offering valuable insights into LLM training. → DCLM is trained on 2.5T tokens using the DataComp-LM (DCLM) framework, with datasets from Common Crawl. → DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments aimed at improving language models. → The DCLM Framework is designed for future experiments with data curation strategies like deduplication, filtering, and data mixing. Why is this significant? Apple's AI strategy now includes open-sourcing all their models. This shift, alongside their recent on-device AI announcements at WWDC, indicates a move toward smaller models and open AI. The future of AI is looking increasingly open. Resources: Model: https://lnkd.in/grgNegcN Repository: https://lnkd.in/gzcTsxqY Dataset: https://lnkd.in/gqY_U2x5

46

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

Navigating TSV files can be confusing, but it doesn’t have to be. Check out Michał Marcińczuk, PhD's latest article where he breaks down the common issues and show you how to work with TSV in Pandas like a pro. #Pandas #Python

6

1 Comment

Just wrapped up an exciting first week of the MLOps Zoomcamp offered by DataTalksClub, founded and led by Alexey Grigorev! This immersive, free programme is a fantastic way to gain hands-on MLOps experience. We're following along with pre-recorded lectures, building practical skills, and learning best practices and tools for taking machine learning models to production. The experience is enriched with homework, peer reviews, and community learning. In week 1, we covered the foundations of MLOps and the key stages involved in a machine learning project lifecycle. We then got our hands dirty by setting up our development environments (using GitHub Codespaces or provisioning VMs on AWS) and building an ML model - a linear regression model to predict ride durations using the NYC Green Taxi trips dataset. Some of the key learnings from week 1: ✅ Understood the different stages of an ML project: Data preparation, Model training, Model evaluation, Model deployment, Monitoring, and Retraining ✅ Learned best practices for configuring Python environments using Docker and Docker Compose ✅ Practised fetching, preparing, and performing exploratory data analysis on the NYC taxi data ✅ Trained and evaluated a linear regression model using scikit-learn and Python The hands-on approach with weekly homework is a great way to solidify learning. The Week One homework involved building a linear regression model to predict NYC Yellow taxi ride durations. Looking forward to diving into Experiment Tracking in week 2!  Stay tuned. #MLOps #MachineLearning #DataScience #DataTalksClub #ContinuousLearning

10

Carmel Eve shares a detailed investigation into Pandas and Pandas for Spark API, trying to diagnose an out of memory exception and how a change in approach reduced job time from over and hour to 30 seconds. https://lnkd.in/eu3-9mwy

4

I've just written a blog post once again - I find it quite interesting to show some 'agents' I've seen in PyPI. One uses a nice Python's feature to trick static analysers - just execute functions by writing their names in Unicode. Looks like your code editor couldn't load a font with local characters (if you use only English, you've probably never experienced how irritating it is, but it happens constantly on some websites) https://lnkd.in/d3S6wNGC

10

#ITiCSE paper #3: 🔥 Compiler AND runtime errors predicting programming performance. In this paper we compared measures of student struggle based on both compiler and runtime errors. Both types of errors play role in predicting students' final grades.

1

Excited to share and interact with fellow Data Scientists at the Data Science Festival in London on the 18th May! https://lnkd.in/e6HmhQCM Luca Valer and I will be talking about the use of LLMs for Topic Modelling by considering domain specific semantic textual similarity (STS) and the fine tuning of open source LLMs such as Llama2 for multi-class text classification purposes where the context length is longer than that of BERT (512 tokens). #DigitalMarketing #DataScience #GenerativeAI #AI #Informa

18

As part of my growing interest in Language Models (LLM) I stumbled upon structured generation, or how to constrain LLM output and wanted to share my hands-on experience with the Linkedin community . Structured generation enables the use of tools with LLMs and LLM agents, which is next on my study roadmap. For now, I've written this article about creating a visual knowledge graph for books using LLMs and Python plotting. If you have time, please take a look at these visual summaries of classic books: - A Tale of Two Cities https://lnkd.in/eEzffEBS - Dream of the Red Chamber https://lnkd.in/eVJpYGmx - The Alchemyst https://lnkd.in/e5Z3Yb_J - The Little Prince https://lnkd.in/eEgUDXyk The details about how I created those pretty graphs arein this article https://lnkd.in/eXrWbA8N. Looking forward to hear your guys feedback on those simple yet pretty neat applications of LLMs #AI #LLM #Literature #DataVisualization #Innovation

6

3 Comments

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments