Joshua Hartman

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

43

5 Comments

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

Good detailed overview of Retrieval-Augmented Generation (RAG), one of leading techniques to improve accuracy of LLMs. When developing a production-grade LLM-based system, it's essential to understand the advantages and limitations of different techniques (RAG, prompt engineering, fine-tuning etc), across training and serving/inference phases, to improve LLMs for specific applications (see my previous post https://lnkd.in/giuJdJJt) A balanced approach to LLM applications that takes into account accuracy, latency, and cost is crucial for successful productization at scale. #artificialintelligence #largelanguagemodels #GenerativeAI #airevolution

10

New Best Practice on CPT 👏🏻 Continued Pre-Training (CPT) refers to the cost effective alternative to pre-training for a new, specific domain (eg. a medical domain) that was not well represented in a model’s pre-training corpus. In this blog, we outline the entire CPT process for: Showing how to optimize hyperparameters. Measuring the effect of different datasets. Developing heuristics for mixing datasets. Mitigating forgetting.

14

We've been playing with o-1 models for last few days (via APIs + ChatGPT), and absorbing the technical paper. Our initial view: * o-1 models are super impressive at times but disappointing at others. * These are preview versions, hope that the 'production' versions will be significantly more reliable. * Not an improvement in the ladder gpt3.5 - gpt4 - gpt4o, but a new direction / branch like gpt2 --> gpt3 was. The key question which tasks are suited to use o-1 type models? Given the costs and rate limits, these are definitely not suited for anything more than experimentation *at the moment*. Sam Altman said that this is an imp step to make Agentic frameworks possible. Its hard to understand what that really means but I'm assuming that's chaining prompts to execute on a series of tasks based on conditional logic (& reward answers that are objectively correct). Key questions: 1. How does model decide how much time to "think" at inference time? Since it doesn't allow RAG, Function Calling, Browsing, etc., all the "critical thinking" is basically reflecting upon its training "Chains of Thought". Does that mean more inference time will produce better results? 2. Now that "Step-by-Step" is not required in prompting, how should prompts be modified to get the best out of o-1 3. How to interpret the "Show your Thoughts"? OpenAI decided to hide the real CoT (for obvious reasons), and let the model 'summarize' those chains. So, this is like a human trying to "articulate" their "thoughts" in a way that average human being can understand. Is it possible to get more specific "articulation" if the end user is an expert, and can understand nuances much better? This will be huge in legal domain. 4. A bold claim from the technical paper: "Through training, the models learn to refine their thinking process, try different strategies, and recognize their mistakes." The last part is really intriguing. I've not be able to verify (or read a paper which has shown that the model recognized it going down a wrong path and then corrected itself) 5. Harvey has mentioned that its working with OpenAI to develop these models. What does that mean in terms of RLHF'ing transaction or litigation related legal tasks? OpenAI's announcements say that its specifically RLHF'd for coding, math, physics, chemistry, biology. All highly objective areas. (With extra emphasis on safety, red teaming, avoiding jailbreaking, etc.). One of their tests show that o-1 models are slightly worse off than gpt-4o in more subjective areas like personal writing or editing text? Where does that leave highly contextual tasks like transactional review? In our testing (similar prompts), we saw both o-1 mini and preview missing a 'Right of First Proposal' language as an aggressive clause for buyer in a supply agreement (which also has exclusivity clause btw) whereas gpt-4o got it right the first time. We'll share more examples later. #o-1 #llms #ai Richard Tromans Leonard Park

11

5 Comments

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

OpenCoder doesn't get enough love They open-sourced the entire pipeline to create QwenCoder-level code models. This includes: - Large datasets - High-quality models - Eval framework Tons of great lessons and observations in the paper 📝 Paper: arxiv.org/abs/2411.04905 #special #linkedin #Family #Pakistan #skills #islam #government #AI #ML #MachineLearning #tesla #microsoft #openai

5

Distributed incremental Spark-style batch jobs: https://lnkd.in/gTh9SD9P Looking for feedback on this, this was a quick prototype, but with a bit of elbow grease it could be used for big data computation with perfect incrementality, no need for manual serialization, no dependency conflicts, and jobs that deploy in seconds with a function call. cc Unison Computing

22

2 Comments

Amazing performance boost by DeepSpeed. FPx speed-ups on training and inference is always preferable to INTx quantization.

9

🚨 New paper w/ interesting insights for fine-tuning and GPU poor folks 🚨 🚀 Exploring Fine-Tuning for LLMs: Full Fine-Tuning vs. #LoRA - Are They Truly Equivalent? 🤔 As LLMs grow, fine-tuning them for specific tasks becomes challenging due to high computational costs. Recently, Low-Rank Adaptation (LoRA) has become a popular alternative, promising similar performance with fewer parameters. But are LoRA-tuned models the same as fully fine-tuned ones? MIT’s latest study dives deep into this question, revealing some surprising differences! 🔍 📝Key Takeaways: 1. Spectral Differences 🧬: LoRA introduces “intruder dimensions” not found in fully fine-tuned models. These alter the model’s weight structure, suggesting that LoRA and full fine-tuning leverage different parts of the parameter space, even if they hit similar accuracy on specific tasks. 2. Generalization & Robustness 🌐: Models fine-tuned with LoRA may struggle with new or sequential tasks, forgetting previously learned knowledge more easily. Full fine-tuning, however, shows better retention and robustness in shifting environments. 3. Effective Rank & Optimal Performance ⚙️: LoRA’s rank matters! Mid-range ranks (like r=64) perform well on tasks and retain pre-trained knowledge, while lower or higher ranks can lead to greater forgetting. Smart parameter tuning (like setting α=2r) can help LoRA resemble full fine-tuning more closely. 4. Why It Matters 💡: LoRA offers incredible parameter efficiency, but its unique trade-offs might affect its use in scenarios needing high adaptability and long-term memory. With adjustments, LoRA can bridge some of these gaps, but full fine-tuning remains the go-to for complex or multi-tasking demands.

25

1 Comment

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

This is really awesome. I used notebookLM to summarise a recent paper that shows how o1 like model can be reproduced. After providing the pdf from Arxiv, I noticed that the generated podcast was almost 49 minutes long. Very different from the earlier 10-12 minute version. And another impressive feature was the interactive mode where you can interject the two hosts and ask a question which they very graciously respond to. I wanted to share the audio summary but YT didn’t allow me to upload the WAV file. YT expects a video mp4 file. So I found free videos from Pexels and stitched them together with ffmpeg. The videos are meditative acquarium shots of fish swimming around. And with the audio discussing reproducing o1 as an engaging podcast episode, the video keeps the distraction to the minimum. The paper itself is a great read but if you have a 30-40 minute commute, or want to watch some cool fish swimming around and learn about o1, check out

12

Long-term memory is "just" a built-in document store that supports `get`, `put` and `search`. But that simplicity is for a reason. We see that memory is still application specific - and so want to give developers low-level building blocks to use In many ways, memory is similar to agents: - Both are highly hyped - Both are hard to define - For both, we see that application specific implementations (e.g. vertical) are winning

171

7 Comments

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

I just realized: we should start saying "build memoization" rather than "build caching." We aren't really talking about least-recently-used or time-to-live algorithms, we're talking about "don't bother recompiling this piece if none of its inputs changed." (It has been puzzling me that some people are so deeply fearful of "build cache" as a concept, and honestly I think it's mostly because most build systems have non-hermetic state scattered all over the place. Maybe a slightly more precise term for the type of caching we're concerned with would clarify the semantics in question.)

83

9 Comments

Interesting paper showing that LoRA doesn't learn your data that well which raises questions on how effective it is at fine tuning your Gen AI model. LoRA Learns Less and Forgets Less - https://lnkd.in/eG4tEuSe

19