Ron F.

Hey everyone, I am an investor and advisor to Katanemo.... They are reinventing the infrastructure primitives for GenAI and launched Arch - https://lnkd.in/gCrbbkEV - an intelligent prompt gateway. Give the project a look on Github, and star the project if it resonates

88

3 Comments

The session unfortunately wasn't recorded, but here are the slides from my talk at LFMS on quantifying the ROI of contributing to OSS https://lnkd.in/gwdQa3UZ

65

5 Comments

We launched structured data retrieval in Bedrock Knowledge Bases last week at re:Invent. While embeddings-based semantic search has worked wonders for unstructured data, we felt we need a new approach for structured data. We needed a way to translate customer natural language queries to an expressive query language that traditional databases already support. https://lnkd.in/eTPStNif #rag #text2sql #aws #bedrock

29

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

Project update! My application tracking extension is now doing fuzzier matching and injecting new information into LinkedIn (and other sites) to keep me from doubling up on applications. Next up: local storage and bulk import/export operations...

12

2 Comments

Want a preview of what I'm going to be talking about in my keynote for RubyConf? You can catch a glimpse of it here!

10

ModernBERT is an open-source embedding model which represents a significant leap in encoder-only transformer architecture merging: > Long context (8k tokens) > Local-global attention for efficiency > Up-to-date code and text training data (2 trillion tokens) > FlashAttention & Unpadding to push performance on consumer and server GPUs > High retrieval and classification accuracy that sets new benchmarks across GLUE, BEIR, code tasks, and more. Learn more about this exciting new embedding model here: https://lnkd.in/en5ud9VE

2

Performance is key! As we cannot share the detailed work with (former) customers of Rockset (acquired by OpenAI), we decided to implement the Rockbench benchmark to check how streaming ingest between CrateDB and Rockset (acquired by OpenAI) compares. The results are very insightful. We see: ✅ CrateDB achieves 6-9x lower latencies than Rockset for streaming ingest. ✅ CrateDB outperforms Rockset on the same hardware while saving about 20% costs. ✅ When volumes increase, the latency increased linearly in Rockset, while remaining mostly flat in CrateDB. As a plus: 1️⃣ Achieve full high availability due to the distributed nature of CrateDB (benchmark results are achieved including replication in CrateDB) 2️⃣ Achieve higher throughput on the same amount of CPUs as Rockset (at 50% of memory needs) 3️⃣ Full open-source solution that prevents you from market shifts On side note: If you are thinking to switch to Elastic/OpenSearch, this benchmark has been created to compare Rockset vs. Elastic, so CrateDB is also faster than Elastic for streaming ingest cases! Read the full comparison here: https://lnkd.in/d_nEMDkx

29

1 Comment

The Site Reliability Engineering: How Google Runs Production System book is arguably one of the most prevalent and well-known SRE resources out there, but there are many other great options for folks looking to expand on their learning. We recently spoke to Google's Reliability Advocate, Steve McGhee, in our Humans of Reliability interview series. In addition to his interesting anecdotes on the early days of SRE at Google, and his journey to becoming a Reliability Advocate, he also shared a handful of his favorite SRE resources, which we compiled here into a list: https://lnkd.in/gpUVhyRa

16

This is the upside of having a monorepo. Successes like this depend on being able to change code at scale safely and with clear understanding of performance impact. The changes themselves are done by a fairly small team, but the systems that enable us to continuously roll changes with critical company wide impact like this are key to make this possible.

38

9 Comments

I just realized: we should start saying "build memoization" rather than "build caching." We aren't really talking about least-recently-used or time-to-live algorithms, we're talking about "don't bother recompiling this piece if none of its inputs changed." (It has been puzzling me that some people are so deeply fearful of "build cache" as a concept, and honestly I think it's mostly because most build systems have non-hermetic state scattered all over the place. Maybe a slightly more precise term for the type of caching we're concerned with would clarify the semantics in question.)

83

9 Comments

There's still time to participate in our 2025 State of Open Source Report. For everyone I know working with OSS take a few minutes to tell us what your organization is using and why. Here's a link to the survey: https://lnkd.in/gS_h-xYh Thanks for participating and helping us put together a great report!

10

1 Comment

That's wild 🤯. Meta & Google is still on mono repo in 2024. I suppose it's more cost effective for both companies to re-engineer Mercurial & Perforce (respectively as "Sapling" & "Piper") & build custom development tooling, than to change their workflow for all of their engineers. In the case of Meta: a standard git merge operation would have taken 45 minutes. Legends has it, they consulted the Git team & was told to break up their repo. So instead, they went to the Mercurial team & got a better reception. Here's a perspective from one of their engineer. But wait... 🤔 the source code for Sapling & Piper + all of its IDEs & build tools are (probably) not store in the same repo? For sure with Meta Sappling, it's not (you can find it on GitHub). Same story with Android. These repos have totally separate concern domains 🤓 I get it. So maybe, it's "mono-repos" & :. truly, no actual mono repo 😅 With the collective wisdom of the Meta & Google engineering teams (combined!), who am I to judge 🫠

5

1 Comment

I've been quite occupied this week, but I haven't forgotten to summarize the key takeaways from last week's Kubecon 24 held in Salt Lake City. Both I and the team enjoyed the excellent chances to interact and exchange ideas with numerous Kubernetes innovators and open source advocates from all over the world. Notably, our team successfully presented two interesting talks regarding Bytedance's open source project Kubewharf, specifically focusing on the Godel scheduler and Katalyst resource management aspects. These talks have generated much influence within the community. Huge thanks to Lintong Jiang, Yue 'Abby' Y., Jia Deng & @Mingmeng Luo for their great delivery of the talks. Please check out the following links for the details: - Godel scheduler (Bing Li, Yue Yin, Lintong Jiang): Talk link - https://lnkd.in/gmXsgGFY; Github: https://lnkd.in/gR-z32xk - Service profiling & Katalyst (Jia Deng, Cong Xu, Mingmeng Luo): Talk link - https://lnkd.in/gHN-Q33x; Github: https://lnkd.in/gZ4FTZ8V Thanks to the feedback from the community: https://lnkd.in/gDXqipRZ It's so rewarding to have conversations with our internal open source advocates, namely Ting Zou (Tina Tsou), He Cao, James Xiang, Ye Xu. Particularly Ting Zou (Tina Tsou) has provided useful connections on how to interact with the community and also rich ideas on how to improve our contribution & influence back to the community. Additionally, I had the opportunity to meet external K8s open source innovators like Yin Ding from Google, Wei Huang from Apple, Richard Sikang Bian from Ants Group, Chen Wang from IBM/Redhat, and a dozen of people from LinkedIn, CoreWeave, Datadog, and many others. I was so grateful for the wonderful time spent sharing your insights on all the fascinating k8s topics. Looking ahead, we'll aim to make even greater open source contributions to give back to the remarkable K8s community. #kubecon #CloudNativeCon #CloudNative #Kubernetes #K8s

48

sharpening writing skill is one of killer skill!

2

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

What does it feel like to be an engineering manager at Amazon: Since the RTO announcement it became popular to write shit about Amazon, but it's still one of the best companies around the world. A few weeks ago, Gilad Naor shared with the 'Leading Developers' readers 3 lessons from his time as an engineering manager at Amazon. Here is the first one, read the full article here: https://lnkd.in/dHBxv2Ha 𝟭. 𝗦𝗼𝗹𝘃𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺𝘀 𝘄𝗶𝘁𝗵 𝗺𝗲𝗰𝗵𝗮𝗻𝗶𝘀𝗺𝘀 My first meeting at Amazon was baffling. I entered a room with about a dozen Software Development Managers (SDMs) and Product Managers (PMs). Everyone sat in silence and looked at their laptops or the printed papers they had before them. It was my second day at Amazon, and I picked up one of the extra sheets and started reading. After those first ten awkward minutes, everyone finished reading and the meeting began. Why do Amazon meetings always start this way? This is a specific example of how Amazon approaches problem-solving. It is the "Good Intentions Don't Matter" approach. In a famous internal video, Jeff Bezos talks about why good intentions are irrelevant. When things don't work, it does not matter what people wanted. All that matters is fixing the problems with mechanisms. For example, many busy managers come to meetings without first reading the prep material. They all intended to read the 6-Pager before the meeting. They just got sidetracked by the latest emergency. Scolding the managers won't solve the problem. A mechanism will. This is why all Amazon meetings start with a study period. The remaining meeting time is then spent on high-quality decision-making. Meta solves the same problem by creating cultural norms. I can't count the number of times that I heard a senior leader cut off the presenter: "Stop, assume that everyone here already read the slides. Let's get to the point." This is a culture-first approach. I have seen this work well in some teams, and not work at all in other teams. At Amazon, the solution is part of the company's "operating system". ------------------------- Anton here - by now everyone is familiar with the 6-pager concept, but that "operating system" thinking can be used almost anywhere. Think about something annoying that your developers do, and how can you solve it. For example, I remember being annoyed that people don’t update the Jira tickets. Finally, after requesting multiple times, now we just take 20 seconds from the standup to update them.

56

8 Comments

A super clear, straight forward example of how to do cost effective RAG using LangChain for your document pipeline and Groq for compute resources. The code demonstrates how to speed up RAG and reduce costs using quantized embeddings (reducing 32 bit values to 8) Nice job Sahibpreet Singh! https://lnkd.in/gf5QMCKK

44

2 Comments