Milad Shahidi

Wonderful article here from Dhanush Kishore on the statistical engine we have developed at Skyscanner to evaluate A/B tests and estimate uncertainty:

70

2 Comments

Recently I've been playing with "Swarm Intelligence", wondering in the back of my mind how I might apply it to my "day job" and merge with modern methods in the age of transformers...etc. Today I came across a paper (https://lnkd.in/g_bUdGB7) in the TL;DR newsletter that uses many inferences from small models to generate much better solutions than individuals, and at 3x lower cost than using a large model. And it is merely generating several samples independently without any interaction among the models. What if we instead applied a swarm intelligence algorithm to this approach so that inference N was informed by all the prior inferences? Would we better navigate the exploit/explore trade-off? This is the sort of place my video series will be heading. Relevant quote from the abstract: "...when it comes to inference, we often limit the amount of compute to only one attempt per problem. Here, we explore inference compute as another axis for scaling by increasing the number of generated samples. Across multiple tasks and models, we observe that coverage - the fraction of problems solved by any attempt - scales with the number of samples over four orders of magnitude. In domains like coding and formal proofs, where all answers can be automatically verified, these increases in coverage directly translate into improved performance. When we apply repeated sampling to SWE-bench Lite, the fraction of issues solved with DeepSeek-V2-Coder-Instruct increases from 15.9% with one sample to 56% with 250 samples, outperforming the single-attempt state-of-the-art of 43% which uses more capable frontier models."

50

3 Comments

Lots of genAI makes me eye roll so hard I risk tearing my optic nerve. ML has its place, GenML only looks like it has a place. On the one hand I should keep quiet and land a fat pay day. On the other I have a sense of ethics and want the people paying me to benefit from the work I do. There is a reason the number one item on our ML project check list is a cost benefit analysis.

4

This is a major development for the open-source AI ecosystem. Apple has gone fully open-source! They have just released DCLM, a 7B open-source LLM, including weights, training code, and dataset. They plan to open-source all their models for deployment across their billion devices. Apple's embrace of open-source marks a significant shift. Here’s a summary: → Apple has released and open-sourced their DCLM 7B model, outperforming Mistral/Qwen2/Gemma on various benchmarks. → It is primarily trained on English data with a 2048 context window, and licensed under the Apple Sample Code License. → The release includes an exceptional explanation of the data curation process, offering valuable insights into LLM training. → DCLM is trained on 2.5T tokens using the DataComp-LM (DCLM) framework, with datasets from Common Crawl. → DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments aimed at improving language models. → The DCLM Framework is designed for future experiments with data curation strategies like deduplication, filtering, and data mixing. Why is this significant? Apple's AI strategy now includes open-sourcing all their models. This shift, alongside their recent on-device AI announcements at WWDC, indicates a move toward smaller models and open AI. The future of AI is looking increasingly open. Resources: Model: https://lnkd.in/grgNegcN Repository: https://lnkd.in/gzcTsxqY Dataset: https://lnkd.in/gqY_U2x5

46

Phi-3 Vision is now available on Huggingface. Phi series models from MS have smaller sizes but higher accuracy in their size range because the models are trained on a curated high quality dataset. "The model is intended for broad commercial and research use in English. The model provides uses for general purpose AI systems and applications with visual and text input capabilities which require: >> memory/compute constrained environments; >> latency bound scenarios; >> general image understanding; >> OCR; >> chart and table understanding. Our model is designed to accelerate research on efficient language and multimodal models, for use as a building block for generative AI powered features." Mithilesh Thakkar

9

"We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench)." https://lnkd.in/g9mSgaRC).

7

It took too much time but the 4th and last part of my "Various aspects of working with time series data" article series is finally out. 🎉 In this part I am describing different time series tasks and the approaches to solve them, and I also give a brief review of the new models that were recently introduced and may improve time series task performance significantly. LMK what you think. 💭 Cheers! 🍻 https://lnkd.in/d3USjgbC #datascience #timeseriesdata #machinelearning #deeplearning #AI

27

2 Comments

Are you working with time series data and want to boost your forecasting game?📈 My latest blog post, "Unlocking Time Series Insights: Why Stationarity and Autocorrelation Matter?" dives into the fundamental concepts every data enthusiast needs to know: ✨ How stationarity simplifies analysis and sharpens your models. ✨ Why autocorrelation is the key to understanding patterns over time. ✨ What white noise tells us about randomness. Whether you’re a beginner or a pro, this post breaks down complex ideas into practical takeaways. Ready to unlock your time series potential? 👉 Read the full post here https://lnkd.in/e-Kgb-KQ

15

Such an interesting read.😍🤯 This builds on anthropic's earlier research on mechanistic interpretability (Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.) Some things that stood out to me: - Social biases can be mapped to feature spaces - By steering these feature spaces we can actually change the social biases in an llm - Steering can sometimes lead to loss in model capabilities. - They find what's called a steering sweet spot which -5 to 5 in steering feature scale where one can change the bias of the model without much change in model capabilities. - Steering a certrain feature are sometimes unpredictable and can extend to other contexts. such as one bias steering can actually effect another bias. - They even found what's called a "neutrality feature" which consistently reduced bias in multiple categories. - This study used a 34M Sparse auto encoder but they suggest that scaling this can lead to better feature sensitivity which makes sense. This is such nuanced research but i'm grateful to the team at Anthropic for pursuing this. I believe this might be one of the most impactful works in the long run. https://lnkd.in/eXeruJkq

8

On fairness of AI or ML algorithms indeed, haven't thought much from that angle before // Simply removing the "protected attribute" is insufficient. As long as the model takes in features that are correlated with, say, gender or race, avoiding mentioning it will do little good. // // Redundant encodings - simply omitting the protected attribute may make it impossible to measure model bias and also mitigate it // quoted from "The Alignment Problem" by Brian Christian

9

2 Comments

Food is more than just nourishment; it's deeply intertwined with our health, lifestyle, and culture. In the age of AI, eCommerce search is evolving from a tool into a source of inspiration. It's no longer just about showing you what you're looking for—it also offers delightful surprises you didn't know you wanted. Often, we don't know exactly what to add to our cart. Maybe we have a vague idea, a theme, or an impulse—craving traditional Asian cuisine, trying making sushi at home but needing guidance on the ingredients, planning a fun football night, aiming for a healthier lifestyle with gluten-free meals, or simply wondering, "What's for dinner?" Thanks to LLMs and their vast knowledge of brands, cuisines, recipes, nutrition, cultures, and more, we're making these experiences possible. It takes a dedicated team to seamlessly blend the world knowledge of LLMs with Instacart's insight into our users and product catalog. This involves careful engineering, thoughtful product design, and rigorous A/B testing with real-world traffic. It's been an exciting journey collaborating with such a talented team to supercharge eCommerce search with LLMs.

6

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

Continuing with the geometric theme in blogs. Next up: Transformers 📐✨ Takeaways ✅ 📏 Convex Polytopes: Attention in transformers results in the mapping of data into convex hulls with long contexts => larger polytopes. 🔑 Hashing Functions: MLP partitions are fixed. However, the input data to the partitions are dynamically transformed based on context. 🎯 Prompting Strategies: Discover the implications of geometric insights on open-ended vs descriptive prompting. Understanding the geometry of LLMs isn't just an academic exercise – it has practical applications! Get informed in your approach to prompting. 🧠💡 🔗 Article here: https://lnkd.in/gCjdqBMD #AI #LLM #PromptEngineering #Geometry #Transformers

10

1 Comment

The next part of my Causal AI series is up on Towards Data Science, this time exploring how Linear Programming can be used to optimise treatment strategies: https://lnkd.in/eb2ErFgi

46

My 2nd blog post for the Vizuara Substack is live now This blog has been written in a lot of detail about the Embeddings in NLP: Firstly, we went through the evolution of approaches to work with texts. - Bag of words - tf-idf -Word2Vec -Transformers and sentence Embeddings. Then, we discussed how to understand whether texts have similar meanings to each other. After that, we saw different approaches to text embedding visualisation. Here is the link to the blog post: https://lnkd.in/gn3YGZqM Raj Abhijit Dandekar #nlp #embeddings #vectors #word2vec #tf-idf #bagofwords #sentenceembeddings

32

1 Comment

Really happy to share that our work on “Learning Metrics that Maximise Power for Accelerated A/B-tests” will appear in the Applied Data Science track of #KDD2024. We address a common problem in industry: how can we minimise type-II errors in online controlled experiments? Can we simply learn a metric combination that directly minimises this quantity? This project started out as “let’s implement the ideas from this great paper by Eugene Kharitonov et al.”, but we ended up proposing some extensions to their seminal work that seem to be quite effective, and are now used across ShareChat. Proud to have this work out there, let us know what you think. Additionally, we were assigned an Area Chair that clearly cared: initial low-confidence reviews prompted them to invite more reviewers (we got 8!), and the AC themselves correctly grilled our rebuttals to point out arguments they deemed insufficient. It’s easy to be positive about peer review when your paper gets accepted, but it seems there are still some unsung heroes out there... Work with Aleksei U.. ACM SIGKDD & Annual KDD Conference

116

9 Comments

Following on from last week's article, this week I cover Activation Functions

1

Today I’m reading about visual perception in the context of data visualization from a wonderful book by Kieran Healy (link below). Did you know the following? - We are attracted to edges in visual media, seeing contrasts and brightness relatively rather than in the absolute. - We can see edge contrasts better in monochrome than color. - Perception lacks uniformity across a span of shades of brightness and possible colors. - A phenomenon called pop-out makes elements in a graph easier to see or find.  Pop-out on a color channel can be stronger than on a shape channel. - We look for structure in our graphics, even finding it in random data.  The strong inferences we make around relationships based on sparse information are called “gestalt rules.” Why is this important for those of us creating data visualizations in any language or tool? - Background choices matter - the same color or shade will be perceived differently against a dark background vs. a lighter one. - Introducing colors vs. monochrome may complicate a graph and make it harder to perceive structures. - When picking colors for our graphics, we want ranges that are not only numerically uniform, but also perceptually.  I just read that the R language has color models and color spaces that attempt to provide this!  - Take care with color, shape, size, angle, elongation, etc. (channels as Prof. Healy describes it) as some have stronger perception than others (see graphic I made in R below to compare shape and color channels). - Given time, we will find structure even in random data, so don’t forget about adding in statistical modeling to your workflows. Book: https://a.co/d/0v2i99I Accompanying R package: https://lnkd.in/gaDCWUCa Professor Healy provides references to the extensive research that backs this up in his excellent book.

13

Excited to share and interact with fellow Data Scientists at the Data Science Festival in London on the 18th May! https://lnkd.in/e6HmhQCM Luca Valer and I will be talking about the use of LLMs for Topic Modelling by considering domain specific semantic textual similarity (STS) and the fine tuning of open source LLMs such as Llama2 for multi-class text classification purposes where the context length is longer than that of BERT (512 tokens). #DigitalMarketing #DataScience #GenerativeAI #AI #Informa

18