Girish Sarveiya

Thanks People+ai! Check out the video below to know what I am up to these days! I am building hardware to make AI accessible to all!

59

5 Comments

Today as part of the panel on “Technological challenges to enable metaverse” we provided a broad perspective on the Metaverse enablement covering software, applications, E2E cloud computing infrastructure and Reliability. Thank you to the co-panelist Jyotika Athavale Dejan Milojicic and Balasaravanan (Bala) Thoravi Kumaravel for providing the in depth insight into these areas and promoting meaningful conversations into the metaverse enablement #metaverse #2024 IEEE Symposium on Mixed and Augment Reality #IEEE Metaverse

135

Improve your LLM training throughput by 20% and reduce memory usage by 60% with a single line of code change! Checkout the insightful engineering blog on Liger-Kernel developed by LinkedIn! Kudos Pin-Lun (Byron) Hsu, Yun Dai and team!

19

I’m really excited about Broadcom Inc. They just hit a record revenue of $51.6 billion for fiscal year 2024, marking an impressive 44% year-over-year growth. What really stands out is their AI revenue, which skyrocketed by 220%, jumping from $3.8 billion last year to $12.2 billion this year—now making up 41% of their semiconductor revenue! Plus, they returned $22 billion to investors in dividends, up 45% from last year. This shows they’re not just focused on growth but also dedicated to rewarding their shareholders. I’m betting on Broadcom, along with Meta and Alphabet. These companies are at the forefront of tech innovation, and I can’t wait to see where they go next! #Broadcom #Earnings #AI #TechInnovation #Meta #Alphabet

1

TSMC Leads the AI Surge, Apple Faces Headwinds, and NVIDIA Steps Up A recent discussion about TSMC’s and Foxconn’s strong 2024 performance was driven by AI momentum. We also explored Apple’s challenges with tariffs, declining market share, and the lack of clarity around the potential of Apple Intelligence in China. Finally, I shared my views on NVIDIA’s latest announcements at CES 2025 in Las Vegas. The tech sector is on track for more transformation, fueled by robust AI demand and innovation. #AI #Semiconductors #NVIDIA #Apple #Foxconn #TSMC #CES2025 #TechnologyTrends #DigitalTransformation

35

It is very important to reduce KV cache memory consumption during long context inference. We introduce ThinK, a method that exploits substantial redundancy across the channel dimension of the KV cache. Our findings reveal that even KV caches optimized by eviction methods, such as SnapKV and H2O, it can still benefit from pruning approximately 40 to 50% of their key cache channels. Our approach results in achieving over a 20% reduction in KV cache memory compared with SnapKV and H2O. Paper: https://lnkd.in/gxpeVdC9

155

https://lnkd.in/gSekNjEX One request I always get from academic researchers is the opportunity to understand and research on real-world product traces. The blog post from Phitchaya Mangpo Phothilimthana and Saurabh Kadekodi has some nice information on synthesized storage IO traces that we released today at Google. (The methodology is also quite cool, if you haven't seen it before, I would look at the Google ASPLOS'24 paper on Thesios.)

223

1 Comment

𝗖𝗮𝗻 𝗚𝗼𝗼𝗴𝗹𝗲’𝘀 𝗻𝗲𝘄 𝗧𝗿𝗶𝗹𝗹𝗶𝘂𝗺 𝗰𝗵𝗶𝗽 𝗰𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝗡𝘃𝗶𝗱𝗶𝗮’𝘀 𝗱𝗼𝗺𝗶𝗻𝗮𝗻𝗰𝗲 𝗶𝗻 𝘁𝗵𝗲 𝗔𝗜 𝗱𝗮𝘁𝗮 𝗰𝗲𝗻𝘁𝗲𝗿 𝗺𝗮𝗿𝗸𝗲𝘁? Google has unveiled its sixth-generation AI data center chip, Trillium, which is nearly five times faster than its predecessor. This new chip positions Google as a strong contender in the AI data center market, dominated by Nvidia. While Google has gradually captured a significant market share, it offers its chips through its cloud platform rather than direct sales. The Trillium and Nvidia’s Blackwell chips are expected to be operational by late 2024, intensifying the competition in the AI chip market. #𝗮𝗶𝗰𝗵𝗶𝗽𝘀 #𝗱𝗮𝘁𝗮𝗰𝗲𝗻𝘁𝗲𝗿 #𝘁𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝘆 #𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻

48

"What good is building great technology if you can't communicate it to the world?", the MIT admissions officer scrawled on the margin of an applicant's essay. I had scored a cool student summer job there, filing applications in vast cabinets. Which is a pre-2000, pre-Internet story for another time. Read Amin Vahdat's article–a Google VP of Core ML–on the history of TPUs, or Tensor Processing Units. I would like to help explain the use cases for TPUs vs GPUs, and the underlying technology that enables these use cases. /☀️ TECH DISRUPTION THESIS You have to grok the Infrastructure layer to understand how technological disruption happens–and that's something that few venture capitalists understand. That it's 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 that powers 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺𝘀 that power 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀. Knowing infra helps you predict which applications will be disruptive with a technological paradigm shift. ☀️/ TPUs are designed to solve problems in domains, e.g.: 🚨 Real-time applications–requiring high throughput and low latency, e.g. language translation, image recognition. 🚨 Data center efficiency–where it's critical to deliver scalable, efficient machine learning services. 🚨 Deep Learning model training–because TPUs handle DL tensor operations, like in large-scale applications involving CNNs and RNNs. Now you may see why Google applications like Vertex AI, Search and Translate require massive advancements in AI chips, i.e. TPUs. The technological underpinning of TPUs vs GPUs are tied to (a) instruction set architecture (ISA) and that (b) TPUs excel in scenarios where TensorFlow, the Google ML framework, is used. More on the software later, here's an Infrastructure explainer: 🚨 TPUs do specialized tensor operations–TPUs have an instruction set that focuses on accelerating tensor operations, such as matrix multiplications and convolutions, which are core to neural network computations, and fundamental to machine learning, especially deep learning. 🚨TPUs are optimized for and integrated with TensorFlow–They include custom hardware, such as systolic arrays, that accelerates TensorFlow operations, making them highly efficient for specific machine learning workloads. The tech underpinning are different for GPUs: 🚨 GPUs, unlike CPUs, which are optimized for sequential processing, can handle many operations simultaneously. This makes them great for tasks that can be broken down into smaller, independent operations—like rendering millions of pixels on a screen. Hence, NVIDIA's headstart in Graphics, and later, for Computer Vision, processing large datasets in parallel. 🚨GPUs have a more flexible ISA that supports a range of operations, making them good for general-purpose parallel processing, such as graphics rendering, scientific simulations, and ML applications. 🎁 "You have to dive deep to the ocean floor to see the real beauty of the seas." --Vik Chaudhary, 2024

21

6 Comments

🛠️ 𝐎𝐋𝐀𝐏, 𝐎𝐋𝐓𝐏, 𝐃𝐚𝐭𝐚 𝐋𝐚𝐤𝐞, 𝐑𝐚𝐭𝐞 𝐋𝐢𝐦𝐢𝐭𝐢𝐧𝐠 𝐚𝐧𝐝 𝐏𝐮𝐛/𝐒𝐮𝐛 — #𝐌𝐮𝐬𝐭𝐊𝐧𝐨𝐰𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠𝐓𝐞𝐫𝐦𝐬 𝐏𝐚𝐫𝐭 #9 🛠️ In this ninth part of the series, we explore five fundamental concepts that power modern system architectures, enabling scalability, data-driven decision-making, and reliable communication. Whether you’re designing real-time systems or managing massive datasets, these terms are essential to understand. Here’s a clear, beginner-friendly breakdown 👇: 𝐎𝐋𝐀𝐏 (𝐎𝐧𝐥𝐢𝐧𝐞 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐚𝐥 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠): OLAP systems are designed for querying and analyzing large datasets to generate business insights. ➡️ Powers dashboards, reporting, and data analytics for strategic decision-making. ➡️ Data is aggregated, pre-processed, and stored in multi-dimensional cubes for fast analytical queries. ➡️ Tools like Snowflake, Google BigQuery, or Amazon Redshift are used for OLAP workloads. 𝐎𝐋𝐓𝐏 (𝐎𝐧𝐥𝐢𝐧𝐞 𝐓𝐫𝐚𝐧𝐬𝐚𝐜𝐭𝐢𝐨𝐧 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠): OLTP systems handle real-time, high-frequency transactional workloads, like inserting, updating, or deleting data. ➡️ Used in operational systems like e-commerce, banking, and ticketing where speed and accuracy are critical. ➡️ Transactions are executed with ACID properties (Atomicity, Consistency, Isolation, Durability) for reliability. ➡️ Relational databases like MySQL, PostgreSQL, and Oracle are ideal for OLTP use cases. 𝐃𝐚𝐭𝐚 𝐋𝐚𝐤𝐞: A data lake is a centralized repository that stores raw, unstructured, semi-structured, or structured data at scale. ➡️ Supports diverse data types (e.g., logs, images, tables) for analytics, machine learning, and advanced processing. ➡️ Data is ingested in its native format (e.g., JSON, CSV, Parquet) and processed later as needed. ➡️ Cloud-based solutions like AWS S3, Azure Data Lake, or Google Cloud Storage are commonly used for building data lakes. 𝐑𝐚𝐭𝐞 𝐋𝐢𝐦𝐢𝐭𝐢𝐧𝐠: Rate limiting controls how many requests a user or system can make to an API or service within a given time frame. ➡️ Prevents system overloads, protects against abuse (e.g., DoS attacks), and ensures fair resource usage. ➡️ Requests exceeding the set limit (e.g., 100 requests/minute) are either delayed, throttled, or rejected with an error code (e.g., HTTP 429). ➡️ Tools like AWS API Gateway, Nginx, or libraries like Token Bucket implement rate limiting. 𝐏𝐮𝐛/𝐒𝐮𝐛 (𝐏𝐮𝐛𝐥𝐢𝐬𝐡-𝐒𝐮𝐛𝐬𝐜𝐫𝐢𝐛𝐞): Publish-Subscribe is a messaging pattern where senders (publishers) broadcast messages to topics, and receivers (subscribers) consume those messages asynchronously. ➡️ Decouples communication between services and enables scalable, real-time message processing. ➡️ Publishers send events to a message broker (e.g., Kafka, RabbitMQ), and subscribers listening to topics receive the events they’re subscribed to. ➡️ Systems like Apache Kafka, Google Pub/Sub, or AWS SNS implement Pub/Sub.

10

2 Comments

The session at re:Invent “how Netflix benchmarks Foundation Models (FMs) and LLMs across hardware chipsets” by Amit Arora and Amer Ather highlighted some critical considerations for model benchmarking, including inference server configurations, resource optimization, and performance metrics across different hardware chipsets. Benchmarking and evaluating models becomes essential when users take their applications external facing. When this happens, three aspects become the top of mind for them: Inference at scale, cost to serve and model accuracy. To choose the right model for your generative AI workload, it is essential to target all three vectors: Performance efficiency (latency optimizations, token throughput), Cost optimization (dollar cost per transaction for a generative AI workload that serves a given number of users) and model accuracy (either on domain specific or open source data). In the session, FMBench is highlighted as a tool that targets all three vectors above and provides the most optimal recommendations that users can streamline their genAI decision making with. FMBench offers a comprehensive feature set, offering customization through YAML files, support for multiple inference engines (Triton, vLLM, TensorRT, DJL, HF TGI), deployment support for AWS silicon/GPU/CPU hardware and built-in analytics capabilities. What stands out is the flexibility it offers in benchmarking any model (this includes custom, fine tuned models), on open source/custom datasets, and evaluate those in a scalable way using a panel of LLM judges. You can use this tool today for your use case and benchmark for your generative AI requirements. Here are some helpful resources to get started: 1. Introductory Videos (https://lnkd.in/eXSTHeKd, https://lnkd.in/egzqbSa8): These are introductory/set up videos that contain information on FMBench, how to set it up and a walkthrough of a sample report. 2. FMBench Website (https://lnkd.in/eaEEXAPr): This website is fully searchable to ease common tasks such as installation, finding the right config file, benchmarking on various hosting platforms (EC2, EKS, Bedrock, Neuron, Docker), model evaluation, etc. 3. FMBench GitHub Repository (https://lnkd.in/eM8mmHrJ): The FMBench GitHub repository is open source. Get started with benchmarking and evaluating for your use cases today and reach out if you have any questions or ideas! Manju Prasad #AWS #MachineLearning #Benchmarking #GenerativeAI

141

8 Comments

Hey everyone, I am an investor and advisor to Katanemo.... They are reinventing the infrastructure primitives for GenAI and launched Arch - https://lnkd.in/gCrbbkEV - an intelligent prompt gateway. Give the project a look on Github, and star the project if it resonates

88

3 Comments

Glad to have shared my thoughts around software, hardware, and our insatiable thirst for more energy with Scott McGrew on NBC Bay Area. https://lnkd.in/gYR-q9gS As data centers, and the energy to run them, represent a larger portion of enterprise P&L we need to start thinking about how we're building software, and whether it is efficient, not simply to make an app faster, but to spend the least amount of energy necessary. We're in a new era of tech companies buying nuclear reactors, it's time to get serious about leveraging more efficient hardware to get the most done with the fewest watts. Voltron Data #SQL on #GPUs

51

2 Comments

I look forward to speaking about how quantum processor networking will revolutionize data centers and supercomputers at #sc24

35

3 Comments

Venkat Tummalapalli and I were lucky enough to visit with one of our favorite customers this week. Team Nesh has been working with IEEE over the 2 years to enhance their #AskIEEE solution. #AskIEEE leverages Nesh’s GenAI capabilities to provide accurate and contextually rich answers to queries of IEEE’s vast library of technical literature in engineering and science. With Nesh fueling #AskIEEE, the organization’s 421,000 members gain streamlined access to the information they need from IEEE’s wealth of scientific content. This week, we met with the IEEE team in Toronto at their Board of Directors Meeting to showcase the benefits of a solution like #AskIEEE. Here are some of my takeaways from our conversations: 1. AskIEEE and Nesh have melded the concern of content distribution behind a paywall with the need for GenAI-powered productivity. 2. AskIEE helps IEEE’s global membership work seamlessly across timelines — enabling them to access the niche-knowledge they need even when other collaborators aren’t online. 3. And as always — GenAI enthusiasm continues. Companies and organizations should focus on meeting this enthusiasm from their members and employees. Start with defining a narrow use cases that can scale once proven. Great things ahead for #AskIEEE, the IEEE team Priscilla Amalraj, CAE, Rakesh Kumar, Steve Woods, Sampath Heragu, and Nesh! #GenAI #GenerativeAI #AI #ArtificialIntelligence #IEEE

46

1 Comment

AI GPU clusters with one million GPUs are planned for 2027 — Broadcom says three AI supercomputers are in the works #datacenter #ai #finance #digitalinfrastructure #IndustrialComputing #CloudComputing #AI #EdgeComputing #QuantumComputing

8

2 Comments

Here is everything that I found interesting last month about irresponsible AI. #blog #engineering #leadership #qfm QFM012: Irresponsible AI Reading List March 2024 https://lnkd.in/e-u-Jyzw

7

1 Comment