Sachin Panemangalore

With Google’s recent announcement about its Willow quantum computing chip, which reportedly solves in five minutes a problem that would take the world’s fastest supercomputers ten septillion years (10,000,000,000,000,000,000,000,000), how significant do you think the threat of quantum computing is to Bitcoin and other blockchain networks? Additionally, what is the expected timescale for this threat to materialize?

2

Hey everyone, I am an investor and advisor to Katanemo.... They are reinventing the infrastructure primitives for GenAI and launched Arch - https://lnkd.in/gCrbbkEV - an intelligent prompt gateway. Give the project a look on Github, and star the project if it resonates

88

3 Comments

Can KV cache optimization go beyond just reducing memory footprint? Our latest work, SwiftKV, does exactly that—cutting prefill computation by half and significantly accelerating enterprise inference workloads! We’ve open-sourced the SwiftKV-optimized model checkpoints for the Llama 3 family and vLLM implementations, enabling efficient inference and making it easier than ever to integrate these advancements into your applications. - Tech blog: https://lnkd.in/gZbPBum3 - Paper: https://lnkd.in/g__9NpBd We hope SwiftKV inspires new possibilities for enterprise LLM efficiency for you!

463

6 Comments

Enhancing Java security with Post-Quantum Crypto resistant algorithms :-) https://lnkd.in/gUZEhbh7 https://lnkd.in/gWa3QMBa "Java for the AI world"

26

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

43

5 Comments

I am incredibly excited to announce Arch gateway https://lnkd.in/ePAf-VK8, an open source project for developers to build genai apps faster. It is engineered with purpose-built LLMs, Arch handles the critical but undifferentiated tasks related to the handling and processing of prompts, including detecting and rejecting jailbreak attempts, intelligently calling “backend” APIs to fulfill the user’s request represented in a prompt, routing to and offering disaster recovery between upstream LLMs, and managing the observability of prompts and LLM interactions in a centralized way - all outside business logic. Here are some additional key details of the project, * Built on top of Envoy and is written in rust. It runs alongside application servers, and uses Envoy's proven HTTP management and scalability features to handle traffic related to prompts and LLMs. * Function calling for fast agentic and RAG apps. Engineered with purpose-built fast LLMs to handle fast, cost-effective, and accurate prompt-based tasks like function/API calling, and parameter extraction from prompts. * Prompt guardrails to prevent jailbreak attempts and ensure safe user interactions without writing a single line of code. * Manages LLM calls, offering smart retries, automatic cutover, and resilient upstream connections for continuous availability. * Uses the W3C Trace Context standard to enable complete request tracing across applications, ensuring compatibility with observability tools, and provides metrics to monitor latency, token usage, and error rates, helping optimize AI application performance. This is our first release, and would love to build alongside the community. We are just getting started on reinventing what we could do at the networking layer for prompts. Do check it out on GitHub at https://lnkd.in/ePAf-VK8 Please leave a comment or feedback here or in GH and I will be happy to answer. #gateway #envoy #rust #genai #developers #ai cc Salman Paracha | Muhammad Ikramul Haq | Syed Anas Hashmi | Fahim ul Haq | Farheen Omar | Umar Farooq | Arslan Ahmad | Abdul Rafay Abbasi |

60

9 Comments

FlashInfer v0.2 is here, a lot of technical upgrades, great news for anyone are doing custom attention improvement or just benefit it implicitly from integration of LLM frameworks

39

Improve your LLM training throughput by 20% and reduce memory usage by 60% with a single line of code change! Checkout the insightful engineering blog on Liger-Kernel developed by LinkedIn! Kudos Pin-Lun (Byron) Hsu, Yun Dai and team!

19

Unlocking new possibilities in AI with my latest blog post: "Phi-3 ONNX: Leveraging the Power of ONNX Runtime to Run SLM on CPU." This is a follow-up to my previous post on using Small Language Models (SLMs) with Azure App Service. In this new post, I dive into how the ONNX Runtime enables Phi-3 to efficiently run on CPU, providing a robust and cost-effective solution for building intelligent applications. This blog covers essential insights on making advanced AI more accessible. Check it out https://lnkd.in/ghAuknDs

23

Long-term memory is "just" a built-in document store that supports `get`, `put` and `search`. But that simplicity is for a reason. We see that memory is still application specific - and so want to give developers low-level building blocks to use In many ways, memory is similar to agents: - Both are highly hyped - Both are hard to define - For both, we see that application specific implementations (e.g. vertical) are winning

171

7 Comments

Today I hosted #llama3 for the first time using TWO A100-80GB with 32 cpu cores and more than 150G RAM and it was pretty fast. It costed me $5/hr though. I hope to run a quantized version on one A100-80G (for 16fp one should be enough). I also hosted 8B version on something like A6000 RTX with with only 6 cpu core and around 55G RAM which was quite fast. One question is that is it better to run a 8B-32fp or 70B-8bit! #ai #selfhosting #gpu #llm Some comments: For running llama3, first you need to take permission. Second, The whole gpu on-demand allocation and starting and running takes somewhere between 30-60 min, so depending on your machine cost, you are paying for some of that preparation time.

19

4 Comments

Ok. I have done the #llama3-70b with 8bit precision by #ollama which requires half the RAM, ie, ~71GB . I ran on a system with 2 A6000RTX on FluidStack with 12 core cpu and 110 GB of RAM for hourly rate of $1.6/hr. I did not notice any degradation in accuracy, however, I have not seen any useful benchmark to say for sure. I realized that the best pragmatic metric is to compare price of generating 1M tokens. Here is one example: System with 2 A6000RTX generate 8 token/sec so it takes 3-4 sec to generate a single line of 20 words or so. In an hour you can generate at most 3600*8 ~ 29000 token so the cost of generating 1M tokens is like 35*$1.6/hr= $56/hr which is the twice as expensive of #gpt4-tubro. With that $5/hr 2 A100-80G with 60 cpus and 240GB of RAM, the speed was 50 token per second. So, the price for 1M token was around $27.5 which is much cheaper than the previous system. One last word: System comprised of 2 A100-80G ($20K each) with 60 cpus and 240GB of RAM costs at least $60K with one year operation costs no less than $80-$100K and you can rent it for $35K/yr which is pretty good deal IMO.

3

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

70

3 Comments

A super clear, straight forward example of how to do cost effective RAG using LangChain for your document pipeline and Groq for compute resources. The code demonstrates how to speed up RAG and reduce costs using quantized embeddings (reducing 32 bit values to 8) Nice job Sahibpreet Singh! https://lnkd.in/gf5QMCKK

43

2 Comments

With the o3 model making a massive leap forward—are we so close to achieving AGI that OpenAI no longer sees it as the ultimate destination? o3 scored 87.5% on the ARC-AGI benchmark (high compute) and 76.5% on standard compute. It took four years to progress from 0% (GPT-3) to 5% (GPT-4o), but only six months to leap to an impressive 87.5%. This isn’t just incremental progress—it’s a groundbreaking transformation. Meanwhile, the Financial Times reports that OpenAI is considering removing its AGI clause with Microsoft to unlock more funding. Why would OpenAI move away from a clause tied to its core mission? AGI is hard to define and even harder to measure. If AGI means solving ANY problem a human can easily handle, we’re still far from that point. But what’s happening right now might be just as significant. With models like o3, we’re entering the era of sub-AGIs—systems that outperform humans in specific domains and can execute end-to-end tasks autonomously. These systems have the potential for massive economic impact. As for OpenAI, scaling this kind of research requires billions in R&D. Just three months after the o1 model, OpenAI announced o3. The need for significant funding is clear, and the shift from chasing unclear AGI ambitions to delivering high-performing, adaptable systems may now be the focus. The pace of AI advancement is truly remarkable!

62

1 Comment

A couple of recent college graduates from India against AI finetuning giants like Anyscale and Run:ai? I met Arko C at a NASSCOM event last October. He was building Xylem AI (YC S24) , a platform that allows you to train and deploy LLMs in production. So naturally most Indian VCs rejected him. Because how could a team of three recent college grads compete with the likes SV startups like Anyscale that have raised hundreds of millions of dollars, have teams of experienced AI researchers and a stash of H100 GPUs? Arko hustled and closed two Fortune 500 companies for paid contracts. I personally know unicorns that haven't managed to do this. He then got into YC in the last round in May. There's a lot to still be proved out, but Arko exemplifies how valuable grit is in building startups. I’m doing a webinar with him tomorrow on challenges of using LLMs in production. Link to register in the comments. Disclaimer: I'm a small investor in Xylem.

153

5 Comments

ModernBERT is an open-source embedding model which represents a significant leap in encoder-only transformer architecture merging: > Long context (8k tokens) > Local-global attention for efficiency > Up-to-date code and text training data (2 trillion tokens) > FlashAttention & Unpadding to push performance on consumer and server GPUs > High retrieval and classification accuracy that sets new benchmarks across GLUE, BEIR, code tasks, and more. Learn more about this exciting new embedding model here: https://lnkd.in/en5ud9VE

2