Hans van Dam’s Post

In this two-minute demonstration, I'll be showing the Google Gemma, Microsoft Phi3, and Meta LLAMA3 models, all running locally on my personal computer. You will be able to observe the impressive speed at which these models operate on my system, and see how this performance compares to that of ChatGPT (GPT4). My PC setup includes an AMD Ryzen 7 5800 processor, 32GB of RAM, and an NVIDIA GeForce 3060 with 12GB of GPU RAM. For software, I'm utilizing Ollama, Docker, and Open WebUI. #gemma #phi3 #llama3 #openwebui #ollama

WE NEED TO STOP OpenAI Matthew here can explain it better than I can, but in summary OpenAI should not be allowed to push a regulation on our graphics cards. They are trying to do Regulatory capture which will kill open source. Imagine having to pay a fee or sign up to government or company to use your own Ai model on your own PC. https://lnkd.in/g6FeejjA

I'm going to nerd out for a moment: this GPU is fucking cool! Last week, I bought an HP Omen gaming computer (on sale, thankfully) because it had the highest Bang for your Buck ratio (yes, that is a mathematical thing I invented.) It came with an NVIDIA GeForce RTX 4060Ti and it's the first non-Google Colab GPU I have ever used. Had a hell of a time figuring out how to use it, but finally...FINALLY...set it up and Python recognized it. I built a denoising convolutional autoencoder. classifier (DCAC) using Tensorflow about a year ago, but I am an idiot and can't get my GPU to work with it now. Doing what any reasonable person would do, I switched to PyTorch and it fucking works! My DCAC was refactored for PyTorch and I ran a wide variety of experiments today. I'll post the results soon!

Nvidia just launched Chat with RTX 1) AI chatbot that runs locally on your PC. Nvidia just released "Chat with RTX", which lets you create a custom GPT chatbot trained on your own local data - including documents, videos, and more! 2) Custom GPT chatbot You can create your own custom GPT chatbot with this tool. This tool represents NVIDIA's push towards enabling developers and users to harness the power of generative AI locally on their PCs. 3) Data is the new gold Since "Chat with RTX" runs locally, without needing an internet connection. This ensures that the user's data remains private. you cn download from here https://lnkd.in/dJwuc6iq system required RAM 16GB or greater OS Windows 11 Driver 535.11 or later File Size 35 GB #code #nvidia #rtx #ai

“ChatRTX is a demo app that lets you personalize a GPT large language model (LLM) connected to your own content—docs, notes, images, or other data. Leveraging retrieval-augmented generation (RAG), TensorRT-LLM, and RTX acceleration, you can query a custom chatbot to quickly get contextually relevant answers. And because it all runs locally on your Windows RTX PC or workstation, you’ll get fast and secure results.” The GPU (Graphics Processing Unit) is certainly not for everyone: NVIDIA GeForce™ RTX 30 or 40 Series GPU or NVIDIA RTX™ Ampere or Ada Generation GPU with at least 8GB of VRAM It installs a web server and Python instance on your PC, which then leverages Mistral or Llama 2, then use Nvidia’s Tensor cores on an RTX GPU to speed up queries. Read my next post as well on another locally run generative AI chatbot. #ChatRTX #GenerativAI #LLM #LocalAI #Privacy

Llama 3.1 has just been released, and it's impressive. Llama 3.1 405b matches much larger models, the 70b outperforms GPT-4o, and the 8b almost matches GPT-3.5. Llama 3.1 8b outshines the model that powered ChatGPT in late 2022 and can run on my laptop. And the 405b model? Sure, you can run it on your laptop too. You just need 231 GB of disk space. No biggie, right? 😅 I tried to run the 405b on Ollama Docker on my Dell 5530 Ubuntu laptop with Nvidia GPU. Given the hardware limitations, I'm not sure this will work well, and just like in the picture, it's still downloading...

NVIDIA released a demo of a RAG Model and I found the deployment of it is so fascinating since chat with RTX uses retrieval-augmented generation (RAG), NVIDIA TensorRT-LLM software and NVIDIA RTX acceleration to bring generative AI capabilities to local, GeForce-powered Windows PCs. Users can quickly, easily connect local files on a PC as a dataset to an open-source large language model like Mistral or Llama 2, enabling queries for quick, contextually relevant answers. #AI #LLM #RTX #NVIDIA #RAG Resource: https://lnkd.in/dMi8xiJR

StableDiffusion running at 45fps on my laptop GPU 🤯🫨 Unbelievable speed boost! 🚀 Taking the integration of #vvvv and Python to the next level, I've now enhanced the library to push the performance boundaries even further. With the same seamless object marshalling, this upgraded setup boasts a 50% speed increase, achieving a smooth 45 FPS! 😱 This video showcases a hyper-optimized #StreamDiffusion with #TensorRT, turning the PyTorch tensors directly into shader-ready textures. This minimizes overhead and maximizes the creative potential in real-time #AI art generation. Experience the power of vvvv combined with cutting-edge AI, all running on a RTX 4090 mobile GPU—comparable to a RTX 4070 Ti desktop, but now even faster than in my previous video. The future of interactive installations and creative coding looks bright! 😎 Stay tuned for more progress updates... #madewithvvvv #stablediffusion #touchdesigner #processing #creativecoding #realtimeAI

NVIDIA's text-to-image model SANA is now licensed as Apache 2.0. > Generates images up to 4096 × 4096 res. > Sana-0.6B is competitive with Flux-12B, being 20 times smaller and 100+ times faster. > Deploy Sana-0.6B on 16GB GPU, <1 second to generate a 1024 × 1024 image. SANA (Code is Apache 2.0; Model weights are custom licensed: NSCL v2-custom) enables high-definition content creation at very-low cost! Use it with their official gradio app: https://nv-sana.mit.edu/ SANA is from NVIDIA's NVLabs. Quick start guide with gradio 🤠: https://lnkd.in/gSjXFAyg

I built an XCP-ng (FOSS fork of Xen) home server with two rtx3090 GPUs in 2022. At the time, NeoX-20B was the largest GPT-style model available for local inference, and I think Tim Dettmers had either just taken over the BitsAndBytes project from Meta (or it had just been open sourced, something like that) so quantizing weights wasn't as straightforward. I don't think it was even integrated into Transformers yet. The XCP rig has served (npi) me well over the past two years. It wasn't trivial to set up (e.g. PCI passthrough for not-quite-enterprise mobos can involve kernel stuff) but it's been trivial to maintain ever since. And Xen's specialized host OS approach makes it very easy to clone, revert, snapshot, etc. which has greatly simplified all downstream tasks, including the identical start environments I use for each iteration of my ML container repo: https://lnkd.in/gvm9wCGS This Jackalope footage was generated in a minute or two using one of the rtx 3090s. Local txt2vid has already surpassed Gen-2, which I could have sworn was still really impressive earlier this year (mostly because Sora wasn't and still isn't available for public use). There's also two implementations of Pixtral used to explain the contents of images, and Phi-3.5-vision is used to describe some IKEA PDFs that are scraped using a selenium docker container searching duckduckgo. And I put one langchain RAG example that downloads, chunks, and semantically searches War & Peace for a specific passage. And there's a function-calling llama3.1 example for generating recurrence rule ("rrule") strings from natural language. But, if you just want to make lots of jackalopes, I understand that too. The perceptron (aka McCulloch–Pitts neuron, according to wikipedia) is slowly and inevitably absorbing all prior domains of media into common multimodal latent spaces. This obviously includes centuries-old franken-taxidermy prank cryptids.