Alex To

OpenCoder doesn't get enough love They open-sourced the entire pipeline to create QwenCoder-level code models. This includes: - Large datasets - High-quality models - Eval framework Tons of great lessons and observations in the paper 📝 Paper: arxiv.org/abs/2411.04905 #special #linkedin #Family #Pakistan #skills #islam #government #AI #ML #MachineLearning #tesla #microsoft #openai

5

Amazing performance boost by DeepSpeed. FPx speed-ups on training and inference is always preferable to INTx quantization.

9

What if your OpenAI API calls were 100% reliable? This is the claim from the Structured Outputs announcement yesterday. Here is the excerpt: "On our evals of complex JSON schema following, our new model gpt-4o-2024-08-06 with Structured Outputs scores a perfect 100%. In comparison, gpt-4-0613 scores less than 40%." I have been using libraries like instructor to get structured output, but this might mean that all you need is the API now. What do you think? Link: https://buff.ly/46SD2Cd

2

People rightly state that there is a lot of hype surrounding #AI. This reflects our obsession with applying new tools to *every* problem, fit for purpose or not. 😄 Here's the good part, though: when applied properly, AI is a huge force multiplier. Imagine a lever ^ 1000000. I've been sharing, any time I have the opportunity, the "good parts": areas in which AI can have real utility now. Not someday. Not maybe. Absolutely yes, and now. One area in particular is #RAG, Retrieval Augmented Generation. RAG == Turning an AI loose on YOUR DATA. Imagine all of those documents in which vital information is squirreled away, hard to search, even harder to correlate *across* those myriad documents. What a treasure trove if only you could leverage it! With RAG, you can...and pretty easily, too. Using AI and RAG, 1+1 pretty quickly adds up to 500. Or more. Even now, the power of your AI-enabled application is more limited by your imagination than the capabilities of the tools. If you want to know more, get just a taste of what you can do with AI using RAG, join me this Monday at #RAGHack. See below for more details and to save your spot! https://lnkd.in/gYAsiSP4

13

1 Comment

In context of debate if large context #LLM models will eventually negate the need for #RAG, an article from Databricks has some good insights - • Longer contexts boost RAG accuracy... to a point • Performance often peaks at 32k-64k tokens, then declines • Top models (GPT-4, Claude 3.5) maintain consistency at scale • Others show unique failure modes (e.g., copyright concerns, summarization instead of Q&A) • Optimal context size varies by model and task • Lack of long-context training data may explain some issues Key takeaway: Long context and RAG are synergistic, but LLMs need refinement to fully leverage extended contexts. https://lnkd.in/eehdKtCm

17

I can sense the LLM-hype cycle crashing soon. Don’t forget: 1. End-user value matters. Just adding LLM doesn’t make your product better 2. LLMs can’t plan/reason; no amount of monkey patching will overcome this limitation 3. RAG isn’t always better than vanilla BM25 search processed 200+ applications for the AI hackhouse. Many seem to not understand the basic limitations of LLMs + everyone seems to be building similar wrappers on openAI API. I think it’s a function of perverse incentives: it is profitable to hype up your product’s capabilities Ultimately, users don’t care about AI or LLMs. What matters is if you’re solving their problems in a significantly better way v/s all other alternatives. (And remember now all other alternatives are also LLM-powered).

176

15 Comments

This is really awesome. I used notebookLM to summarise a recent paper that shows how o1 like model can be reproduced. After providing the pdf from Arxiv, I noticed that the generated podcast was almost 49 minutes long. Very different from the earlier 10-12 minute version. And another impressive feature was the interactive mode where you can interject the two hosts and ask a question which they very graciously respond to. I wanted to share the audio summary but YT didn’t allow me to upload the WAV file. YT expects a video mp4 file. So I found free videos from Pexels and stitched them together with ffmpeg. The videos are meditative acquarium shots of fish swimming around. And with the audio discussing reproducing o1 as an engaging podcast episode, the video keeps the distraction to the minimum. The paper itself is a great read but if you have a 30-40 minute commute, or want to watch some cool fish swimming around and learn about o1, check out

12

A very useful checklist to follow when you have to put in production Transformer based models

5

New post! Probably the most desired enhancement for database privilege management is finally here in 23ai. #database #sql https://lnkd.in/gXkXGWWH

31

🚀 Breaking News: Cerebras Systems Inference Gets 3x Faster! Llama 3.1-70B at 2,100 tokens / second⚡️  Here's what this means:  ⚡️ 16x faster than the fastest GPU solution  💪 8x faster than GPUs running Llama3.1-3B (a model 23x smaller!)  🎯 Equivalent to a full GPU generation upgrade (H100/A100) in one software release 🌐Real-World Applications Already Transforming Industries: 🧬 GSK (quote from Kim Branson, SVP of AI/ML): "Cerebras' inference speed is fundamentally improving our drug discovery process through innovative AI applications and intelligent research agents." 🎙️ LiveKit (powering ChatGPT's voice mode): "With Cerebras Inference, what was once the slowest stage in the pipeline is now the fastest" - Russ d'Sa, CEO 🛠️ Technical Improvements Under the Hood: Rewritten & optimized critical kernels (MatMul, reduce/broadcast, element-wise ops) Asynchronous wafer IO computation Advanced speculative decoding implementation Maintained 16-bit original weights precision Verified accuracy by Artificial Analysis ⚡️ What This Means for AI Development: Instantaneous voice & video processing Advanced reasoning capabilities Real-time intelligent applications 10x more reasoning steps without latency increase Enhanced multi-step agentic workflows  Experience the Speed Yourself: Try Cerebras Inference via chat or API at inference.cerebras.ai #LLM #Inference #GenAI

74

1 Comment

Databricks is seizing the LLM tooling market 👾 The last significant mention of Databricks in the LLM realm was their release of Dolly last year. At that time, we had Llama 🦙 from Meta, but its licensing was not truly free, or at least I remember some discussion around it, so Dolly 🐑 (apparently, hoofed fluffy animals were the theme?) briefly filled that gap. It had been trained on a custom instruction fine-tuning dataset by Databricks and was released with a commercial use license. Commercial use is still very much Databricks' focus, it seems. Dolly was quickly overshadowed by “fully” open-source models, with new ones released nearly every week. However, what was missing were tools that made deploying LLMs and integrating them into existing infrastructures 🛜🖥️ at companies easy... Thus, Databricks was hard at work, acquiring companies like Mosaic ML (specializing in providing a platform to train and deploy large AI models) and developing new features. Now in various states of preview or release (it's honestly a bit hard to tell even after reading the documentation 🙄): 🔵 vector search - a vector database right next to all the other data tables your company might already have in Databricks, for all your RAG needs or similar 🔵 a marketplace offering (hopefully easy) access to their new foundation model DBRX (yes, they also trained and released that) and various other LLMs for experimentation and deployment 🔵 model serving endpoints for LLMs 🔵 probably more features that I'm forgetting right now (I highly recommend their release post about DBRX and their own documentation for the curious reader) At first, I was a bit sad that DBRX (their new LLM) was completely upstaged by Llama 3 despite great metrics. I listened to a podcast about DBRX with the engineering team and they seemed really cool 😊😅 but in the end, Llama 3 is “just” a model and not a data and ML platform, and it seems convincing enterprise customers that Databricks is a capable partner for LLM use cases is a more important goal for them 🤔 Full disclosure: I haven’t tried out the new LLM features yet - though I have chatted with DBRX a bit on Poe - so they might not be that great (yet), but I thought it was fascinating how they seemed to focus on the tooling market as a strategy. If anyone knows some good resources that discuss this shift towards tooling in the LLM space more, drop them in the comments, I’m always curious 👀

16

2 Comments

#RAG with advanced retrieval technique RAPTOR for long contexts. When working with long-context documents, we cannot just chunk the documents and embed them. Instead we would want to have a good approach for minimalist document splitting for long context LLMs. This is where RAPTOR comes into picture. Recursive Abstractive Processing for Tree Organized Retrieval [RAPTOR] is a new and powerful indexing and retrieving technique for LLM in a comprehensive manner. It adapts a bottom-up approach by clustering and summarizing text segments(chunks) to form a hierarchical tree structure. → The leaves are a set of starting documents. → Leafs are embedded and clustered. → Clusters are then summarized into higher level (more abstract) consolidations of information across similar documents. → This process is done recursively, resulting in a “tree” going from raw docs (leaves) to more abstract summaries. We can apply this at varying scales; leaves can be: → Text chunks from a single doc → Full docs With longer context LLMs, it’s possible to perform this over full documents. This tree structure is key to RAPTOR function as it captures bot high level and detailed aspects of text, which is particularly useful for complex thematic queries and multi-step reasoning in questioning and answering tasks. This process involves segmenting documents into shorter texts called chunks and then embedding the chunks using an embeding model. These embeddings are then clustered by a clustering algorithm. Once clusters are created, the text associated with each cluster is summarized by using a LLM. The summaries generated form nodes in a tree with higher level nodes providing more abstract summaries. Know more about RAPTOR in this research paper: https://lnkd.in/gDgZP6Vj Also, read this article by Plaban Nayak: https://lnkd.in/g-7HpADg ----------------------------------------------------------------- BTW, you also need a robust database in RAG. I recommend using SingleStore Database. Sign up and get $600 worth free credits. Try SingleStore database for free: https://lnkd.in/gCAbwtTC

229

1 Comment

Hallucination is Inevitable in LLMs An important new paper out that shows that even with perfect data and an idealized architecture “LLMs will always hallucinate”. I suspect that this will over time be linked to Gödel’s Incompleteness Theorem [1] and Turing’s Halting Problem [2]. Every technology has trade-offs. We love LLMs but they need to be used in the right way. [1] Good pop science depiction in Oppenheimer (Gödel was Einstein’s walking companion) or also see Rebecca Goldstein’s book Incompleteness [2] Turing is depicted in The Imitation Game and also the progenitor of the famed Turing Test.

36

Glad to see that NGS-IQ's Excel Support has finally caught up with what I've been doing with Excel support in my now free SQL iQuery for the last 10 years. If you're an NGS-IQ user, check out their new features in this itJungle article. https://lnkd.in/geKRGm3Z

9

1 Comment

Usage of managed LLMs and out of the box API providers in early chatgpt induced frenzy era aka just an year ago meant you were prone to LLM drift and prompt drift - meaning the quality of output for the same input had high chances of degradation over time. Do you still see this behaviour with any of the LLM models accessed via managed APIs? Do let me know in the comments! Looking forward to discussing more about the groundlevel/practical pros and cons of self hosted vs managed LLM services at AWS AI Day, Pune today (Image a sneak peak from the presentation deck) #LargeLanguageModels #AI #PromptEngineering

19

1 Comment

Thanks Werner Bogula for the really insightful study. I’m only just getting into model comparison, for now using HuggingChat and some tightly defined use cases. The difference between them is notable, interesting to see, and makes me think the future is probably going to be based around multi-LLM workflows. #genai #multillm

1

What if I told you that a single, tiny scalar parameter in a Large Language Model could determine whether it produces brilliant text—or complete gibberish? This is precisely what researchers at Apple have discovered in their fascinating paper on super weights (link: https://lnkd.in/dvTUiijY). In their study, they found that pruning a single scalar weight—what they call the super weight—in the Llama-7B model completely destroys its ability to generate coherent text. For example: • Before pruning: Llama-7B generates a “reasonable completion.” • After pruning: It produces “complete gibberish.” These super weights, while incredibly rare, are disproportionately critical to a model’s performance. Here’s another striking case study: • Prompt: “Summer is hot. Winter is “ • With the super weight: The model confidently predicts “cold” with an 81.4% probability. • Without the super weight: The model awkwardly guesses “the”, with a low probability of just 9.0%. What Makes Super Weights Special? 1. Stopword Suppression: Super weights ensure models focus on semantically meaningful words instead of common stopwords like “the” or “and.” 2. Super Activations: These weights induce rare, high-magnitude activations that propagate through the network, significantly influencing outputs. 3. Robust Compression: The authors propose a clever quantization technique: preserve super weights at full precision while quantizing the rest. This hardware-friendly, data-free method achieves results competitive with more complex quantization strategies. Why This Research Matters: • Efficiency: Understanding and preserving super weights enables effective compression of LLMs for deployment in resource-constrained environments. • Insight: It highlights that even the most massive models rely heavily on a few critical components. This work by Apple is a great reminder that sometimes, the smallest details make the biggest difference. You can read the full paper here: https://lnkd.in/dvTUiijY.

36

1 Comment