Pushkal Katara

We had a great session about NVIDIA NIMs with Anshul Jindal and Mark Moyou, PhD yesterday. 🚀 From Mark: We got lots of info on inferencing with NIMs and different types of NIMs like embedding models, generative models, reranking models.. 🎨 I walked through how you can use NIMs for embedding, retrieval and generation with #Haystack 🚀 And finally, Anshul did a live coding session where he showed how you can deploy and scale your own self-hosted NIMs. If you missed it, or don't even know what I'm talking about but you wanna learn, the recording is up (adding to the comments👇) All resources, including the example repos and guides are in the description. #AI #LLM #Nvidia #opensource

36

2 Comments

"Prompt Engineering" by Lee Boonstra. This paper covers everything from basic prompting techniques to advanced strategies like Chain of Thought (CoT), Tree of Thoughts (ToT), and ReAct, along with practical tips for code-related prompting. Key sections include: - Configuring output with temperature, top-K, and top-P settings - Using zero-shot, one-shot, and few-shot prompting effectively - System, role, and contextual prompting techniques - Best practices, including experimenting with input formats and adapting prompts to model updates https://lnkd.in/gH5xrivh

13

At Voice of Reason AI, where we focus on interfaceless personal aids for individuals with memory and brain disorders, inference speed is not just a luxury—it’s a necessity. Our AI may remain silent most of the time, yet it must continuously listen, understand, and analyze context so it can respond instantly when help is needed. Think of it like the cymbal player in an orchestra, silent for long stretches but ready to deliver a perfectly timed crash. For us, it's essential that our AI “cymbal” sounds precisely when a user needs guidance. This makes Cerebras' advancements incredibly exciting, as they could enable us to create real-time, ambient aids that seamlessly enhance daily life without unnecessary delays. Such speed is crucial for AI that doesn’t merely react but anticipates needs in real-time, helping individuals maintain independence and dignity.

2

❝ How do you taste your Wine? With the right Glass, served from the right Bottle. How do you utilize your LLM? With the right Task, served from the right Compute environment. ❞ If the quote above caught your interest, check out my blog post that was featured on the Toyota Connected India Medium channel last week: Leveraging LLMs in Cloud - 🔗 https://lnkd.in/gfHsdevH --- I hope this blog adds a bit of amusement for the seasoned AI/ML engineers 🧠👨‍💻 with the carefully crafted images and perhaps even the silly wine analogies 😉. Whether you're a cloud engineer ☁️ 👨‍💻, a product owner of cloud-based apps, or someone new to Generative AI in the cloud, this blog offers a detailed, beginner-friendly look into the selection and deployment of LLMs in cloud environments. --- A heartfelt thanks 🙏 to my colleagues at Toyota Connected India for taking the time to review, shape this piece and even share a creative writeup like below in the original post 🫡 👇 🤔 Curious to dive in? Click the link below to explore the blog on Medium! 🌐 🔗 https://lnkd.in/gfHsdevH

18

Excited to share that our extended abstract titled "Incorporating Foundation Model Priors in Modeling Novel Objects for Robot Instruction Following in Unstructured Environments" has been accepted for presentation at the Workshop on 3D Visual Representations for Robot Manipulation at ICRA 2024! Visit our website for more details: https://lnkd.in/gcmfMJgu Workshop: https://lnkd.in/gA38HrpH 📝 Authors: Moksh Malhotra, Aman Tambi, Sandeep Zachariah, P.V.M. Rao, Rohan Paul 🙏 Special thanks to our project advisors Prof. Rohan Paul and Prof. P.V.M. Rao for their invaluable guidance and support throughout this research journey! In our paper, we tackle the challenge of acquiring object models when a robotic manipulator is sent in unstructured and unknown environments. We proposes a novel approach to utilizes foundation models with RGB-Depth images to acquire high-fidelity semantic 3D model of objects, required for robotic manipulation. We demonstrate our pipeline in simulation as well as real-world scenarios. Looking forward to discussing our research and exchanging ideas with fellow researchers and professionals at the conference. Stay tuned for more updates! #ICRA2024 #Robotics #Research #AI #3DVisualization #RobotManipulation

11

2 Comments

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

386

12 Comments

Predictions based on Machine Learning and Statistics often rely on maximal likelihood, which can trap LLMs in biases inherited from the majority opinions of training data. This problem has been substantially addressed through conditional statistics, allowing for mitigation of biases and preventing the cancellation of minority perspectives. https://lnkd.in/gMXKqwyB

7

Transformers Transformers has been a major milestone in the field of NLP and heavily used in Generative AI. There are 3 variants of transformer based models and they are 1. Encoder-only 2. Decoder-only 3. Encoder-Decoder Encoder only models: These are also called as autoencoders and these are pretrained using a technique called masked language modeling. A text with a random masked token is sent to the model to predict the masked token. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't _____ at the sign, you will get a ticket. The model is expected to predict the token (word) "stop" These models use bi-directional representations of the input to better understand the full context of a token. Examples of encoder models: BERT Family models Decoder only models: These models are called autoregressive models and are pretrained using a technique called causal language modeling. These models predict the next token using the previous tokens. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't ______" The model will still try to predict the word "stop", but only based on previous tokens. These models are used for generative tasks, including question answering. Examples of decoder only models: GPT Family, Falcon, LLama models Encoder decoder models: These models are called sequence to sequence models and the pretraining objective varies from model to model. For example the popular FLAN - T5 uses a consecutive multitoken masking called span corruption. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't _____ _______ the sign, you will get a ticket. The model will try to predict the tokens "stop at" These models are good at translation tasks. Examples: T5 family In all the above explanations we just took one sentence of text. The LLMs you see in the market are trained on huge volumes of text available over the internet. #LLM #encoder #decoder #transformer #genai

3

What a great talk from Swapnil Sayan Saha, on Physics-Aware Tiny Machine Learning! https://lnkd.in/g9g4YWs4 There's exciting times ahead as Physics-aware TinyML is becoming increasingly important in merging the power of machine learning with operating in the physical world. By integrating real-world physics models into ultra-low-power devices, this approach dramatically enhances the accuracy and efficiency of TinyML applications. From improving energy efficiency in smart sensors to enabling more accurate anomaly detection in industrial applications, this approach to machine learning at the edge helps machines better understand their environment - without sacrificing power or performance. As we explore the possibilities, this combination of physics and Edge AI opens up new opportunities in fields like healthcare, environmental monitoring, and autonomous systems. It's a huge leap toward making AI more sustainable and impactful, and allows us to build more useful AI-enabled systems that operate in (and on!) the physical world. #TinyML #EdgeAI #PhysicsAwareAI #AIInnovation #Sustainability #MachineLearning

4

Great explanation and visualization, thanks for sharing Alex Razvant! I would add that other graph compilers have a similar flow, but the differentiation is in the implementation details. For modern AI architectures, graph compiler optimizations are not enough to achieve high performance and utilization, and many AI compilers have dedicated implementations. FlashAttention (kernel-fusion) and PagedAttention (memory management) are examples of dedicated optimization techniques that are not achieved by generic compilation (not to mention Speculative decoding and other algorithmic "tricks"). By the way, this is the main reason in my opinion for why we see so many competing inference engines for LLMs (TensorRT, vLLM, TGI, LMDeploy, ONNX Runtime, SGLang, CTranslate2, llama.cpp, MLC-LLM, DeepSpeed, IPEX-LLM, etc.) 😅

19

1 Comment

Great work here Rohan Sidankar . Thanks to Rohan and Justin, hushh is known for its work in optimizing neural network search through dimensionality reduction, specifically with ViT-B-32 CLIP for various search tasks as we build out our search index. This allows for maintaining recall while reducing vector size, enhancing search efficiency and effectiveness against large datasets like the LAION Benchmark Repository. Dimensionality reduction impacts neural network search by enabling the maintenance of recall while reducing vector size. This optimization, as seen in Rohan Sidankar's work at Hushh with ViT-B-32 CLIP, enhances search efficiency and effectiveness against large datasets like the LAION Benchmark Repository. ViT-B-32 CLIP can enhance search tasks by allowing for dimensionality reduction in neural network search, as demonstrated by Rohan Sidankar's work at Hushh. This approach enables the maintenance of recall while reducing the vector size, which is crucial for efficient and effective search tasks against large datasets like the LAION Benchmark Repository.

11

🔗 Simulate PyTorch Models with Simulink! 🚀 Engineers, unlock new possibilities by simulating PyTorch and other Python-based models directly in Simulink using co-execution blocks. This integration streamlines your workflow, enabling seamless model testing and validation. 🔍 What's Inside: >> Co-execution of Python models in Simulink << >> Enhanced simulation and validation processes << 👉 Dive into the details: https://spr.ly/6041o1cOX #DeepLearning #Simulink #Python #ModelSimulation #Engineering #MATLAB #MAthWorks #PyTorch

9

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

Learn how an image-text multi-modality model can perform image classification, image retrieval, and image captioning #llms #llmops #image #opencv #chatgpt4 #gemini #aiml https://lnkd.in/gJFh9Ruf

1

Our survey titled "Reasoning and Planning with Large Language Models in Code Development" is now available on the Amazon Science website: https://lnkd.in/d8a8H9yQ. This publication accompanies our lecture-style tutorial at KDD 2024 (late August in Barcelona Spain), delving into how LLMs are transforming the code development landscape. Our survey explores LLM's understanding of code patterns, syntax, and semantics to assist developers in tasks ranging from code generation and testing to documentation and interactive reasoning. #KDD2024 #LLM #CodeDevelopment #AI #MachineLearning #AmazonScience

80

3 Comments