Yuanzheng (Ron) Zhu

Very interesting edge-orientation GNN framework from a recent arXiv paper that improves performance over HodgeNets. It seems fairly easy to extend this new model to TDL frameworks with Hodge Laplacians as boundary operators for invariance, equivariance, and fusion pooling: https://lnkd.in/e-vjbtmj

19

2 Comments

Another great paper from Mathieu Desbrun's group. If you're working with graphs, engineering/physics problems, images, or other discrete structures that might have vector-based attributes, this paper can help you smooth, transform, or pool your vector-based data in a mathematically-sound way: https://lnkd.in/ePQgeVJ8

45

6 Comments

Graph theory has a lot to contribute to optimizing graph neural network architecture for improved accuracy and training times. I haven't seen a lot of this in the literature, but a recent paper explores spectral graph properties and the modification of GNNs to incorporate Cayley graph structures to improve GNN performance. There's a lot more that spectral graph/simplicial complex theory, topological data analysis, and other graph-related tools can contribute to optimizing neural network frameworks in general. Here's the paper: https://lnkd.in/ehHVcRjQ Manny Ko, yes, very much Cayley-based tools on this one!

136

10 Comments

I'm excited to share that a new paper from our group has been released on arXiv! (Link: https://lnkd.in/gjAcD22v) The paper proposes a novel quantum algorithm named Markov-Chain Monte Carlo with Sampled-Pairs of Unitaries (MCMC-SPU) for simulating the thermal equilibrium properties of quantum many-body systems at finite temperatures. Compared to previously proposed methods, our approach requires fewer qubits and quantum gates, making it suitable for the early FTQC era. This work is a collaboration with QunaSys. #quantumcomputing #quantum #algorithms #fujitsu

9

Modeling in the sciences is in fact almost always generative modeling. What a BOLD statement to start a paper with here --> 1.1 Motivation One major division in machine learning is generative versus discrimi- native modeling. While in discriminative modeling one aims to learn a predictor given the observations, in generative modeling one aims to solve the more general problem of learning a joint distribution over all the variables. A generative model simulates how the data is generated in the real world. “Modeling” is understood in almost every science as unveiling this generating process by hypothesizing theories and testing these theories through observations. For instance, when meteorologists model the weather they use highly complex partial differential equations to express the underlying physics of the weather. Or when an astronomer models the formation of galaxies s/he encodes in his/her equations of motion the physical laws under which stellar bodies interact. The same is true for biologists, chemists, economists and so on. Modeling in the sciences is in fact almost always generative modeling

1

1 Comment

This is a fascinating paper evaluating DeepMind’s AlphaFold 3 and its approach to predicting ligand docking: https://lnkd.in/gieM2nW5 The authors used a clever method to evaluate AlphaFold, finding that it can make glaring mistakes when placing molecules in binding sites because its model is likely only fitted to the protein’s backbone geometry instead of actually learning the underlying physical interactions. Not that AlphaFold isn’t a breakthrough technology for drug discovery, but this paper highlights that even the most cutting-edge applications of AI are perhaps still best used to generate candidates, but humans still need to perform robust tests to validate them. We’ve been hearing a similar sentiment from nearly all of our users when bringing AI into cloud infrastructure management. Flightcrew has been read-only from the start, using AI to generate insights but requiring a human to sign off before any changes are applied to their infrastructure. The goal is to speed up existing workflows, not create a new point of failure. Drug discovery and managing Kubernetes are completely different applications but we’re all similarly wary of trusting full AI automation and at this rate, likely will be for some time.

19

Here's a very recent development that designs a neural network capable of exactly modeling semialgebraic functions--such as those found in population dynamics, kinematics, and other dynamic systems models from the social and biological sciences (see blog reference for an example). I'm excited to see how this new branch of deep learning evolves and solves complex biological and social modeling challenges: https://lnkd.in/e-3zQ4ye https://lnkd.in/e3iHpCWP Franck Kalala Mutombo, might be a good avenue for new research on more complicated epidemic models.

34

5 Comments

Mitigating hallucinations is one of the more challenging things about working with Large Language Models. This article provides an interesting approach to detecting inaccuracies, and for automating the process.

3

This paper https://lnkd.in/eGaCaD7s shows that calibrating LLMs to the distribution of their training set always leads to hallucination, at a probability lower bounded by Good-Turing frequency estimation. Good-Turing frequency estimation https://lnkd.in/ewrh3RZC was developed to estimate the frequency of discovering a new species. Assuming the discovery of new species follows i.i.d., then its frequency can be reasonably estimated by the historical discovery rate, which is N_1 / N​, where N_1​ is the number of species observed only once (a.k.a., new discoveries), and N is the total number of observations. Now, calibrating LLMs forces them to emit only in-distribution facts (known species) and zeros out the probability of out-of-distribution facts (unknown species). Therefore, calibrated LLMs will hallucinate when the question is OOD, at a rate of N_1 / N​​. And there is no cure, as non-calibrated LLMs can be even worse. #artificialintelligence #machinelearning #deeplearning

208

18 Comments

 A pre-print of my summer intern's recent paper, entitled “Domain decomposition-based coupling of operator inference reduced order models via the Schwarz alternating method”, is available on ArXiv here: https://lnkd.in/g8qZ94CA. The paper presents and evaluates an alternating Schwarz-based approach for coupling together subdomain-local reduced order models constructed via non-intrusive Operator Inference (OpInf) with each other and with subdomain-local full order models (FOMs).

53

🤖 Quantinuum H2 improvements say a lot about quantum advantage. This is a bit of a different approach to evaluation than previously (though if you want a brief breakdown on the technical implementation, check out today’s newsletter). But, this paper from Quantinuum is significant in its ties to the bigger picture. TLDR: Improvements to Quantinuum's H2 quantum processor allow it to operate with up to 56 qubits and high connectivity while maintaining high two-qubit gate fidelity, as demonstrated with random circuit sampling tasks which are known to be challenging for classical computers. Overall, the improvements to H2 widen the gap between classical ability and quantum ability. As a writer and reader within the space, I can tell you that it’s easy to get lost in the increasingly dense zoos of algorithms, error correction codes, minutiae of various updates to modality architectures, etc. As such, I’ve been considering personal benchmarks for evaluating the value of new papers they can be briefly summed up as follows: ⚆ We need to get answers that weren't accessible before (this can include getting "better" answers), and these answers must be tied to practical applications. ⚆ Energy efficiency must be considered -- if we get a better answer at a significantly higher energy cost, was this truly better than the classically “good” answer? (aka practicality in implementation) ⚆ And, the benefit of applying the solution to the practical application must outweigh the energy cost of achieving that solution. Keeping this in mind, Quantinuum set about to improve their existing hardware in terms of connectivity and fidelity. An aside on connectivity because it doesn't always get the recognition it deserves -- connectivity refers to the capabilities of the hardware in terms of the arrangement of qubits and the available gate operations. Higher connectivity means more direct implementation of quantum gates between a pair of qubits. This helps entanglement as it reduces the number of intermediate operations needed and as such, improves fidelity. Higher connectivity → better entanglement → higher fidelity → better hardware. Plus, better entanglement means more difficulty for classical algorithms to replicate. In terms of previously declared KPIs: ⚆ Advantage over classical solvers is present. ⚆ More efficient hardware implies opportunity for strides towards conservative energy use. Ultimately, focusing on crafting better hardware precedes and leads to finding practical applications. It is these incremental improvements that incrementally increase my already present faith in the technology itself. I look forward to seeing what practical applications the Quantinuum teams' efforts lead to -- congratulations to the team! #quantumcomputing #innovation

6

"Are large language models superhuman chemists?" So what these researchers did was make a test -- a benchmark. They made a test of 7,059 chemistry questions, spanning the gamut of chemistry: computational chemistry, physical chemistry, materials science, macromolecular chemistry, electrochemistry, organic chemistry, general chemistry, analytical chemistry, chemical safety, and toxicology. They recruited 41 chemistry experts to carefully validate their test. They devised the test such that it could be evaluated in a completely automated manner. This meant relying on multiple-choice questions rather than open-ended questions more than they wanted to. The test has 6,202 multiple-choice questions and 857 open-ended questions (88% multiple-choice). The open-ended questions had to have parsers written to find numerical answers in the output in order to test them in an automated manner. In addition, they ask the models to say how confident they are in their answers. Before I tell you the ranking, the researchers write: "On the one hand, our findings underline the impressive capabilities of LLMs in the chemical sciences: Leading models outperform domain experts in specific chemistry questions on many topics. On the other hand, there are still striking limitations. For very relevant topics the answers models provide are wrong. On top of that, many models are not able to reliably estimate their own limitations. Yet, the success of the models in our evaluations perhaps also reveals more about the limitations of the exams we use to evaluate models -- and chemistry -- than about the models themselves. For instance, while models perform well on many textbook questions, they struggle with questions that require some more reasoning. Given that the models outperformed the average human in our study, we need to rethink how we teach and examine chemistry. Critical reasoning is increasingly essential, and rote solving of problems or memorization of facts is a domain in which LLMs will continue to outperform humans." "Our findings also highlight the nuanced trade-off between breadth and depth of evaluation frameworks. The analysis of model performance on different topics shows that models' performance varies widely across the subfields they are tested on. However, even within a topic, the performance of models can vary widely depending on the type of question and the reasoning required to answer it." And with that, I'll tell you the rankings. You can log in to their website at ChemBench.org and see the leaderboard any time for the latest rankings. At this moment I am seeing: gpt-4: 0.48 claude2: 0.29 GPT-3.5-Turbo: 0.26 gemini-pro: 0.25 mistral_8x7b: 0.24 text-davinci-003: 0.18 Perplexity 7B Chat: 0.18 galactica_120b: 0.15 Perplexity 7B online: 0.1 fb-llama-70b-chat: 0.05

3

Brain Teaser #11: What mathematical arguments can explain how **Layer Normalization** enhances training stability in large language models (LLMs) and Transformer architectures, particularly in mitigating issues like vanishing and exploding gradients? Answer: Training destabilization in neural networks, particularly due to gradient explosion or vanishing, typically occurs because of the high condition numbers of weight matrices during backpropagation. These large condition numbers lead to unstable gradient scaling, either amplifying gradients excessively or diminishing them, hindering effective learning. How Layer Normalization overcomes and brings stability in training is briefly discussed here: https://lnkd.in/gmWm42tQ #MachineLearning #DeepLearning #Transformers #LargeLanguageModels #LayerNormalization #GradientStability #AIResearch #DataScience #NeuralNetworks #AIInsights

16

5 Comments

Heterogeneous graph data is ubiquitous; different edge relationships and attribute types come up in social/epidemic graphs, molecular/protein graphs, and buyer behavior graphs. This recent paper addresses heterogeny in graph neural network prediction through a topological tool called sheafs, which aggregate properties of edges and vertices according to a pre-defined (or learned) function. Definitely have a look at this recent arXiv paper out of Cambridge if you are working with these types of graph neural networks! https://lnkd.in/eZHHhtuN

47

1 Comment

AI’s Mathematical Myth Busted? I just published a new article on Medium titled “AI’s Mathematical Myth Busted? FrontierMath Leaves LLMs Nearly Empty-Handed.” This deep dive explores the surprising challenges large language models (LLMs) face when tackling advanced mathematical reasoning. The piece examines: • The limitations of current LLMs in mathematical problem-solving • Insights from the FrontierMath benchmark • What this means for the future of AI in STEM Whether you’re passionate about AI, math, or the intersection of the two, this article provides thought-provoking perspectives on where we stand and are headed. Check it out and let me know your thoughts! Let’s discuss: How can we close the gap between computational power and true mathematical reasoning? #ArtificialIntelligence #Mathematics #LLMs #AIResearch #MachineLearning https://lnkd.in/gK9H4QHe

3

3 Comments

Going through the list of papers for #ICML2024 and #ICLR2024 and found some interesting ones that I'll share over the next few days. But here's the big one that got me hooked: Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs Paper: https://lnkd.in/dWkn_d_6 Authors: Yeonhong Park and others. Essentially, the paper proposes: ➡️ Rapid generation of multiple models at different quantization levels → Think about having 2, 3, 4, 5, 6, 7 or 8-bit models always in memory, but with a memory requirement of only an 8-bit model (well, almost, they do need some metadata but its an order of magnitude lower memory requirement). ➡️ Specialized LLM serving engine with the support to serve different quantization levels of the same model. ❓ Why is this important ❓ Imagine the scenario where you have a highly finetuned LLM (or an SLM) that you deploy for your users. As a failsafe, you also need to deploy a secondary LLM that comes into play if the first one goes down. However, as soon as the secondary LLM comes online, the users will notice - it's not a finetuned version of your first model! Or even if you deploy a finetuned version, you end up deploying TWO models, in memory, in parallel. This paper solves this problem - a possible failsafe method for an in-production LLM that failovers to smaller/higher quantization levels based on bandwidth and user requirements. Incredible! For a truly redundant system though this still doesn't work because if the LLM serving engine goes down, all models go down as well. So don't throw away your failsafe LLM just yet, but this might be a handy alternative. Book a free 15 minute call with me: https://lnkd.in/dGnHs_Mn #icml2024 #iclr2024 #llm #nlp #machinelearning #consulting #generativeai #openai #microsoft #iclr #icml #research #paper

12

I use (and like) Edge Impulse as a web-based GUI for teaching applied neural networks and tinyML in my graduate ML class here at UMD, but I wasn't able to find a hardware kit for using line level audio and piezoelectric transducers as sensors quickly and easily (as in, something that you can get a whole class of students up and running with in one semester). I worked with two grad students (Rob Neiburg and Lukas Aanonsen) to design and build a hardware kit based around the Raspberry Pi pico for exactly this purpose. This board allows the user to run real-time, low-power classification tasks on data from line-level audio (e.g. guitar!) or piezo sensors (vibration signal detection) with no tweaking to default EI firmware. It's got a selector switch for either piezo or audio inputs, and three LED outputs to indicate which of three classes the signal fits into - with the option to break out into many more outputs/classes. I should be posting more details soon at my website, and would be happy to share designs for anyone also interested in using this as a teaching tool. #edgeimpulse #deeplearning #tinyML

25

1 Comment